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Digital Reference Services and Centers at the United States 
Department of Education: 

Analysis and Recommendations 

Executive Summary 

Information service Centers at the Department of Education use email-based 
question and answer services (digital reference) to offer information to 
customers. 

The number of customers is growing quickly (ED.gov Centers receive 21,276 
digital reference questions every week). Customer demand is straining 
Departmental resources, and posing difficult decisions to the Specialists who 
answer questions. Policies for answering questions were originally crafted by 
separate offices and while they serve very well for individual constituencies, they 
may not be easily accessible to newly interested populations of users. Specific 
challenges are standardization of processes and formats across offices, and 
support for optimal tracking, archiving, and creation of Frequently-Asked 
Question (FAQ) files across Centers. 

The Department of Education is not alone in facing the challenges of digital 
reference. In a recent conference presentation^ for example, Kresh and Arrett 
reported that the Library of Congress must deal with growing numbers of users 
and proliferating online resources. Further, they noted that the tools required to 
connect users and resources (human intermediaries, catalogs, indexes, and 
bibliographies) are distributed and diverse. 

CENDP, a group of governmental agencies encountered similar problems 
and has met in yearly workshops since 1997. That group suggested^ that (a) a 



^ “Collaborative Digital Reference Service: Providing Library-Quality Reference 
Service Using Digital Technologies” by Diane Kresh and Linda Arret, Library of 
Congress, for the Virtual Reference Desk Conference Reference in the New 
Millennium; The Evolving Role of the Information Professional, October 14, 1999 
Cambridge, Massachusetts. An article based on the presentation is currently at 
press for RUSA. 

^ The CENDI agencies include Defense Technical Information Center (DTIC), 
Department of Energy/Office of Scientific and Technical Information (DOE/OSTI), 
National Aeronautics Space Administration (NASA), National Air Intelligence 
Center (NAIC), National Library of Medicine (NLM) and National Technical 
Information Service. 

® CENDI: Impact of the Internet on Customer Service and Product Development 
Among the CENDI Agencies. Submitted by CENDI Users Education Working 
Group, by Gail Hodge, CENDI Secretariat Information International Associates, 
Inc., Oak Ridge, Tennessee, August 1997. 



study be conducted to find optimal practices in digital reference, and (b) those 
practices be shared across offices. 

For several years the Department of Education has offered digital reference 
services that are well established and committed to customer service. Now the 
Department has advanced beyond other agencies by conducting a major 
research initiative to optimize those services. The research featured methodical 
data gathering, and accomplished: 

• presentation of findings 

• analysis of current processes, procedures and problems 

• suggestions for optimization of information delivery to customers 

• provision of recommendations for policy 

• outlines of software requirements for possible future automation of processes 

• suggestion of training goals for managers and Specialists. 

The findings may be applicable to digital reference services in many 
Government agencies, and are therefore outlined below. 



KEY FINDINGS 

Specialists reported that they encounter problems in almost every aspect 
of answering and tracking customers' question. For example, they find it 
increasingly difficult to: 

• check quality and content of referred answers 

• formalize and share reference lists (of other Specialists and their areas of 
expertise) 

• share and use FAQs and archives 

• keep FAQs accurate, up-to-date and consistent across related resources 

• differentiate among types of questions and answers 

• understand standards and procedures for tracking, archiving and referring 
questions 

• educate customers of varying expertise levels 

• identify customer populations, and priorities for their levels of service. 

Philosophical differences create divergent policies and practices and are 
particularly troublesome in the Centers' quickly-changing environments: A 
fundamental problem is that Specialists disagree with each other, and with 
managers about the identity of the primary customers, and how to best answer 
their questions. 

That problem is compounded because the influx comes not only from the 
Centers' traditional customers, but from new populations, as well. New users 
have varying levels of skill and may need more contexts in their answers. 
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New customers are asking new kinds of questions, and require more 
definition of on-topic questions vs. out-of-scope questions. Out-of-scope 
questions appear in at least sixteen forms and questions may be answered in 
any of nine configuration types. 

In addition, the growing number of online information resources is 
overwhelming Specialists with more information, new interfaces and 
greater expectations for service. These forces combine to create both 
internal and external challenges to the Centers’ answer management. 



MANAGING ANSWERS (INTERNAL PROBLEMS) 

Tracking allows Specialists to monitor the progress of questions sent to 
other information providers and is important to customer services. Tracking at the 
Centers, however, lacks consistency across referents, time, tools and media 
types. Commitment to overcoming these problems varies across Centers and by 
the abilities of Specialists who must work with the systems. 

Archiving is the storage of answers for possible reuse, and presents some 
of the same problems as tracking. Like tracking, archiving is a fragmented 
process that differs from Center to Center, and is constrained by lack of 
standards and inadequate support. 

FAQs are educational tools that are developed from archived answers to 
Frequently Asked Questions. They are currently difficult to find, and are therefore 
underused. The result is wasted resources and continually re-created answers. 

Procedures for tracking, archiving and creating FAQs were developed in 
separated offices each of which selected its own database platform, processes, 
operating system and applications. These "islands" of development create 
disconnects when Centers attempt to share information internally. 

MANAGING ANSWERS (EXTERNAL PROBLEMS) 

The Centers face two primary external influences that dramatically affect 
their customer service performance. 

First, current events trigger increased demand for information. Because of 
their dynamic nature, however, current events allow little time in which to 
coordinate Department-wide answers that are timely, consistent and accurate. 

Second, journalism influences the Centers both formally and informally. 
Formally, professional journalists may write about a Program office and 
disseminate the Center email address without contacting the Center. This 
causes confusion and makes it difficult for Specialists to find the source. 
Informally, non-journalists (parents, students, teachers and researchers) 
cut, paste and electronically dissemination previously published articles. 

The growing interest that is generated by informal dissemination of 
Department information must be served rather than constrained. To do 
that. Specialists and managers must acquire new skills. 
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The results, originally a disparate group of observations, were grouped 
into broad issues and are outlined below. 



ISSUES 

Findings were grouped into underlying issues (see Section 4) and they are 
outlined here: 

1 . Currently there is no one champion to coordinate standards, documents or 
procedures across Centers. 

2. Policies are inconsistent from Center to Center. 

3. Minimal standardization across Centers impairs effectiveness. 

4. Software was developed separately for each working group and imposes 
disconnects between groups when they attempt to exchange information. 

5. Training is informal for Center Specialists and minimal for managers. 

6. Resource sharing is difficult because of inconsistent polices, standards 
and software. 



SUMMARY OF THE RECOMMENDATIONS 

Policy decisions, standards and procedures should be documented and, 
depending upon the degree of detail, included in updated versions of 
documentation at www.ed.gov/internal. 

More specific recommendations are included in Section 5 Recommendations, 
and they advise that the Centers and their managers: 

1 . choose a champion in the CIO's office to procure resources and determine 
the optimal level of centralization (Section 5.1) 

2. select Level 2 centralization to coordinate policy and standards (5.2) 

3. incorporate AskA software into a Department-wide Intranet (5.5) 

4. employ QuIP protocols to enable resource sharing across Centers (5.5) 

5. use checklists to translate policy into actionable items (5.4) 

6. formalize the Frontline Forum and use it to coordinate software specification 
and standardize operations across Centers (5. 5. 1.2.1) 

7. coordinate a "Fast-response" team to provide fast and accurate answers to 
questions about current events, thus preventing traffic spikes in the Centers 

8. research commercially available software packages to determine if they 
support Center processes and procedures (5.6) 

9. create training goals and plans, and decide on implementation mode(s) (5.7) 

10. evaluate daily operations using checklist (5.8) 

1 1. continuously gather and use feedback to upgrade systems and services (5.9). 

These recommendations are designed to better interrelate and redesign 
the ED.gov digital reference Centers in accordance with Educational Excellence 
for All Children Act (1 999): 
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to... promote the sharing of examples of promising practices developed 
under this authority in order to bring effective models to scale (page 76) 
(and) leveraged resources and promoted high quality (page 65). 

As prescribed in the U.S. Department of Education Strategic Plan, 1998- 
2002 , the report provides guidance regarding: 

• training for front-line employees and planning workforce development 

• supporting strategic partners 

• providing seamless service based on optimal technological approaches 

• capturing and incorporating customer feedback, and 

• establishing standards for internal and external operations. 

The recommendations may also serve to inform and support digital reference 
practice at CENDI agencies, the Library of Congress, and other government 
offices that are facing increasing expectations for digital reference service. 



OVERVIEW OF THE REPORT 

The report is presented in five sections and includes the Introduction 
(Section 1) in which the research goals are outlined, and a description of 
Methods (Section 2). Results (Section 3) describes findings from each of the 
three data sources (government documents, personal interviews, and a poll). 
Issues (Section 4) consolidates the findings and groups them. Recommendations 
(Section 5) contains the recommendations of the research team. Section 5 may 
be used as a planning guide for re-designing the Center's processes and 
procedures. Several appendices provide copies of the instruments, and a 
glossary. 
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Digital Reference Services and Centers at the United 
States Department of Education: 

Analysis and Recommendations 



1 INTRODUCTION 



The Department of Education has actively adopted new media to offer 
information to customers. One medium in particular - email on the World Wide 
Web - offers customers unprecedented interactivity, and poses a number of 
challenges for the Department. 

Currently, the Department has established Web-based information 
Centers to describe public programs, grants and initiatives. At these Websites, 
and in various print publications, the Department provides email addresses of 
government employees who can be contacted for more information, thus 
providing direct communication channels between the public and the 
Department. For topics that generate the most interest (e.g. Office for Civil Rights 
or OCR), the Department created larger, Web-based, digital reference Centers '* 
that receive hundreds of email messages per month. 

As the number of customers increases, these email channels and Centers 
are transmitting unprecedented numbers of questions . Increased demand for 
customer service is straining processes, policies, and resource allocation at the 
Department's digital reference customer Centers. 

Provision of the best services possible is mandated in the Departments 
strategic plan (U.S. Department of Education Strategic Plan, 1998-2002 - 
September 1997). In its continuing effort to provide these services, the 
Department contracted with the Virtual Reference Desk research team (at the 
Information Institute of Syracuse) to evaluate the state of email message 
management at six digital reference sites during the calendar year of 1999. The 
goals of the study were to: 

• analyze current processes, procedures and problems 

• suggest solutions for optimization of information delivery to customers 

• provide recommendations for policy 

• outline software requirements for possible future automation of processes 

• suggest training goals for managers and Specialists . 

The research team's evaluation included: 

• document analysis 

• in-depth interviews with six Department message Centers 

• a poll of all ED.gov sites and other, related information service providers. 



Underlined words are defined in Appendix A Glossary. 
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This report is intended to describe plans, tools and applications that support 
the Department's charter "to use technology and electronic networks to create a 
nationwide system that supports interactive information sharing and 
dissemination to improve educational practice" (Educational Excellence for All 
Children Act of 1999, page 65). 

The report is presented in five sections; Introduction, Methods, Results, 
Issues, and Recommendations. 
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2 METHODS 



The research method comprised a three-part approach including 

• document analysis 

• in-depth interviews at six digital reference sites, and 

• a poll of digital reference administrators representing ED.gov and 
related Centers. 



2.1 DOCUMENT ANALYSIS 



Document analysis was conducted to identify the goals and culture of the 
existing ED.gov digital reference Centers. Included were the Educational 
Excellence for All Children Act (1999), the U.S. Department of Education 
Strategic Plan, 1998-2002 (1997), the Overview and Functions. Office of the 
Executive Secretariat (1992); Department of Education's' Customer Corner . July 
26, 1999, Vol. 1, (20)), and other documents available at ED.gov Websites. 



2.2 IN-DEPTH INTERVIEWS 



In-depth interviews were conducted in Washington, D.C. on July 24*^ and 
25*^, 1999 with representatives of the following digital reference Centers: 

1) National Library of Education (NLE) 

(CustomerService@inet.ed.gov Library0inet . ed.gov , 

http : //www . ed . gov/NLE/ ) 

2) Information Resources Center (IRC) 

(uSA LEARN0ed.gov , http://WWW.ed.gOV/offices/OIIA/IRC/) 

3) Allied Technology Group, Inc (Webmaster@inet.ed.gov) 

4) ED Info ( http : / /www. ed. qov/MailinqLists/ ) 

5) OS Executive Secretary 

6) OESE Program. 

Thirteen individuals were interviewed in six interview sessions. Most 
interviews were conducted in groups of two or three interviewees, and the 
discussions were taped. Interview tapes were transcribed and stored in Microsoft 
Word files. Qualog software was used to assign codes categories and retrieve 
coded text. 

Data analysis of the interviews generated a list of issues that were 
investigated on a broader scale in the subsequent poll of digital reference 
Centers 
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2.3 POLL OF ED.GOV DIGITAL REFERENCE CENTERS 



The third method of gathering data was a poll of all digital reference 
Centers in, or linked to the Department. Notification of the poll was sent by email 
to a list of digital reference Centers (see Appendix B Notification of Poll). Poll 
respondents accesses the designated Website and answered demographic 
questions, questions about work processes, and open-ended questions. A copy 
of the polling instrument may be found in Appendix C Polling Instruments 



® Poll participants were identified in the following way: The research team 
scanned www.ed.qov for “mail to” links, (html fields in a Web site) and found 
about 17,000. After duplicates and Webmaster addresses were removed, the 
remaining list contained over 4,000 email addresses, many of which were entry 
points to the Department of Education's online resources (others email 
addresses represented grantees, task force and advisory group members, 
conference attendees, paper authors, and state agencies.). 
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3 RESULTS 



Results are reported for the three methods of data gathering, and include 
discussions of; 

• document analysis 

• in-depth interviews, and 

• a poll of digital reference Center administrators. 



3.1 DOCUMENT ANALYSIS RESULTS 



Customer service isn't just a slogan, it is a necessary focus of our 
organization. \Ne believe that customers should have seamless 
access to information and services and are striving to meet the 
standards we have set for customer service (U.S. Department of 
Education Strategic Plan, 1998-2002 - September 1997). 

Document analysis showed that high-quality customer service is a crucial 
strategic competency for the Department. The U.S. Department of Education 
Strategic Plan, 1998-2002 , for example requires that guidance be provided for: 

• training front-line employees and planning workforce development 

• supporting strategic partners 

• providing seamless service based on optimal technological 
approaches 

• capturing and incorporating customer feedback, and 

• establishment of standards for internal and external operations. 

The Strategic Plan lists objectives designed to correct problems of 
unqualified technical staff, short term fixes, lack of customer service standards 
and centralization of responses to customer inquiries. In response to these 
shortcomings, the Department created goals, including that: 

by 2001 at least 90% of customers, internal and external, will agree that 
ED products, services, and information, including those on the 
Department's Website, are of high quality, timely, and accessible. 

This research study directly contributes to Objectives 4.1, 4.5, and 4.7 of 
the U.S. Department of Education Strategic Plan, 1998-2002 . 

Another document, the US Dept of ED-WWW Policy and Procedures 
ED World Wide Web Policy and Procedures (1998) establishes procedures specific to 
the communication modes of the World Wide Web. That document directs the 
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U.S. Department of Education (ED) to establish standards and guidelines to help 
customers find, retrieve, and use the information they need. 

The Policy and Procedures document, like the Strategic Plan, falls short 
of describing policy for e-mail-based digital reference. The use of email 
addressed in low level documents such as the WWW Server and Site 
Problems Customer Support Process document , but only for use in 
technology administration. For example: 

If you or someone in your principal office notices that the ED-WWW 

Server is down, send an urgent e-mail to webmaster0inet . ed . gov . 

Put the word URGENT (all caps) at the beginning of subject line. 



and; 



If OERI or the ED contractor notices that the ED-WWW Server is 
down, a notice will be sent immediately to an e-mail list to include 
Internet Working Group representatives and interested managers 

Finally, the Educational Excellence for All Children Act of 1999 
recommends that the use of: 

electronic networks to create a nationwide system that supports 
interactive information sharing and dissemination... (Educational 
Excellence for All Children Act of 1999, page 65). 

Like the other documents reviewed for this study, however, the 
Educational Excellence Act does not prescribe policies, standards or procedures. 



3.2 POLL RESULTS 



A poll was conducted of all ED.gov sites and sites to which they are 
linked. The poll prompted respondents for demographic data, referents, and 
trends developing in their Centers. 

Table 1 Polled Populations shows that some notifications were lost, some 
were auto-responses, and some were answered by digital reference workers. 



ED.gov address non-ED.gov address Total 



No response 


442 


1,872 


2,314 


Auto-response 


174 


824 


998 


Human response 


193 


689 


882* 


Total notifications 
sent 


809 


3,385 


4,194 



Sub-totals in bold italic represent the responses that contributed to this report. 

Table 1 Polled Populations 





Poll notifications were emailed to 4,194 potential participants; 809 to 
digital reference Centers operated by the Department of Education, and the 
3,385 to non-ED.gov digital reference to which various ED.gov Centers link. 

Two thousand, three hundred and fourteen (2,314) addressees never 
responded (of which 442 non-responses were ED.gov email addresses, and 
1,872 were from non-ED.gov email addresses). 

A total of 1,880 responses was received by the research team 
998 of which were auto-responses (advising that the poll notification was 
undeliverable because of unknown address, recipient was out of town, on 
vacation, had moved to a new job, or had retired) and 882 human-generated 
responses. 

Of the workers who responded, their responses fell into two group; Table 
2 Poll Responses shows the nature, and number of poll responses received.. 

ED.gov Non-ED.gov 

Nature of Responses Centers Centers Totals 



Declined to participate 


3 


59 


62 


Participated 


190 


630 


820* 


TOTAL RESPONSES 


193 


689 


882 



* Sub-totals in bold italic represent the responses that contributed to this report. 

Table 2 Poll Responses 

The 820 human-generated responses were delivered via two 
communication channels: 802 responded directly to the Web form, and 18 used 
ASCII-based email. Of the 820 responses, 190 were from ED.gov digital 
reference Centers and 630 were from Centers to which various Department sites 
link. (Although this research began by separately analyzing the two sets of 
responses, the results were remarkably similar. Therefore, results are reported 
here for the entire group, rather than by ED.gov vs. non-ED.gov respondents.) 

The poll served two purposes: to verify whether ED.gov email Centers are 
functioning (thus identifying "dead links"), and to gather data from all Centers 
currently in service. Dead links to ED.gov sites were reported to the Office of the 
Director, Resource Sharing and Cooperation. 

Poll data were stored in a Microsoft Access database. The database 
record employed both numerical and textual fields, and required two methods of 
analysis (database fields may be found Appendix D Database Record Structure 
for Poll): First, numerical fields were compared across respondents to look for 
trends and patterns. Second, each open-ended answer was analyzed using 
content analysis 



3.2.1 DATABASE QUERIES (BASED ON NUMERICAL FIELDS) 
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Microsoft Access queries and reports were constructed to compare 
numerical fields in the database. One query, for example, retrieved information 
about how many questions each Center referred out to other Centers. An 
example of this method is Query 1. 

The purpose of Query 1 was to find out how much question referral occurs 
among the polled digital reference Centers. Query 1 retrieved the following fields; 
email address, number of emails answered, number of emails referred, and the 
record's unique identifier. The resulting report showed that approximately 95% of 
respondents refer questions to other Centers. One quarter of that group refer 
over 10% of their question to other Centers. Query 2 showed that over 80% of 
digital reference Centers discard some questions every week. 

Similar queries showed that 

• ED.gov Centers receive in an average week, a total of 21 ,276 digital 
reference questions every week. 

• each ED.gov Center receives on average 123 questions per week, 
and 

• the range spans from 0-500 questions at individual sites. 

Almost half of poll respondents did not know the name of the email 
packages they were using. The remaining half use the following packages almost 
equally: Outlook, Eudora, Netscape email. Exchange and Pine. 



3.2.2. CONTENT ANALYSIS (BASED ON OPEN-ENDED QUESTIONS) 



Open-ended questions (e.g., "characterize what trends (if any) you see 
forming in your service") allowed poll respondents to describe observations not 
anticipated in the polling instrument. 

Content analysis was used to analyze answers to open-ended questions 
and consolidate them into the following list of observations (in order of 
decreasing frequency): 

Email is replacing phones 

Email is replacing faxes 

Email is reducing need for meetings 

There is increased access to programs 

There is increased expectation for speed of response 

There is increased unsolicited mail 

There are increased numbers of customers 

Pressure is increasing to expand Web-based services 

Pressure is increasing linkages with other sites 

There is increased use of the Web to solve problems 

The Web is increasingly used to establish communities 

There is increased use of the Web to convey information 

There is increased use of the Web for retrieving information 

Increasing demand is unsupported by static resource budgets 

There is increasing use of forms over email to ask questions 

There is increasing need for documentation to improve quality 
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There is more need for dedicated email addresses 
Email is leading to development of listservs and collaborative 
environments 

Increasing need exists for policy and procedures to handle request 
centralization and referrals 
Sites need more prepackaged information 
Email is co-occurring with fewer phone interruptions 
Email is co-occurring with less paper 

The number of customers in any given population is increasing 
The number of populations is increasing (students, parents, teachers) 
There is increased geographical scope of questioners 
Power is shifting to the customers as it is incumbent on Specialists 
to answer 

Customers do less preliminary research 

Questions may be obscure, poorly articulated, and require advice 
rather than objective information 
Specialists must gain increased expertise in expanding domains 
Published articles create spikes in customer demand. 

Out of the 362 poll respondents that answered the question "What trends 
do you see developing?", 148 commented that the most important trend was the 
increasing volume of email. 

Of those 148, 23 respondents reported dramatic increases in out-of-scope 
questions. Nine respondents commented that email messages were quickly 
replacing phone calls, and 14 commented that the nature of the questions was 
becoming more specific, sophisticated or contextual. 

A synopsis of the responses is that: 

(a) increasing numbers of customers must be served 

(b) customers have diverse levels or types of expertise 

(c) each customer type needs different training, format and level of content. 

Clearly, the demands on digital reference Centers and Specialists are 
growing not just in number, but in nature. This is a typical outcome of rapid 
technological diffusion among customers and is illustrated in the Human 
Intermediation Curve (Lankes, 1997). 
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The Human Intermediation Curve 
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Figure 1 Human Intermediation Curve 

Figure 1 illustrates the human intermediation curve, and explains that, in 
some systems, the use of human intermediaries increases faster than the use of 
the technology they administer. 

As diffusion of a technology (the vertical axis) occurs over time (the 
horizontal axis) the number of customers grows. But the new customers do not 
come from the same population as the initial customers. In fact, they represent 
four distinct user groups. 

The Core population is the first and most expert set of users. Core 
customers tend be familiar with all aspects of a specific information product. Core 
users would be familiar with the Department, its purposes, products and its 
methods. A core user might ask, "How can I obtain a copy of the U.S. 

Department of Education Strategic Plan, 1998-2002 (1997)?" 

Secondary customers have some knowledge of the Department, but less 
about the topic they are investigating, for example, "Is there any document that 
tells what the Department's goals will be for the next several years?" 

Topical customers are familiar with general agency topic, but need 
increased definition, direction and synthesis, e.g., "I'm writing a report on 
education - are there any special resources that can help me?" 

General public customers posses minimal understanding of the 
Department and its products, and need increased synthesis. Such a customer 
might ask, "What does the Department of Education do?" 

As the diversity of customer types grows, along with technical expertise. 
Specialists must customize the level of content, the sophistication of language 
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and technical instruction for each answer they supply. This customization 
requires additional time and diminishes the number of questions each Specialist 
can answer. Ultimately, these demands will compel consideration of increased 
resource allocation to accomplish scalability in digital reference Centers. That is, 
the Centers must be able to replicate their core competency (human expertise) 
and increase productivity. 

Regarding the ability to customize answers, one respondent expressed 
the concern that: 

...we (humans) function at a much higher level than the technology 

can. But when you bump up the volume (of emailed questions).... 

The limits of human capacity cannot keep pace when we "bump up the 
volume" of digital reference questions. We must therefore create tools that 
support scalability. Tools such as databases, standards and communications will 
be discussed in Section 5 Recommendations. 



3.3 INTERVIEW RESULTS 



In-depth interviews provided three kinds of information. Demographic data 
help describe the interviewees and their Centers. Process data provide a 
description of how they perform digital reference. The third kind of data were 
generated when interviewees were invited to suggest topics about unanticipated 
issues and trends. They are known here as Interviewee-initiated Topics. All three 
kinds of data were gathered using an open-ended, semi-structured interview that 
was useful in capturing unanticipated issues and trends. 



3.3.1 DEMOGRAPHICS DATA 



Demographic data comprise names, titles, email addresses, U.S. postal 
addresses and locations. Those data were used primarily for contacting and 
responding to interviewees, and are available from the authors. 



3.3.2 PROCESS DATA 



Process data were collected to investigate how Specialists collect, track , 
refer, answer and archive digital reference questions and answers. An 
aggregation of those processes is depicted in Figure 2 Digital Reference 
Processes at the Department of Education. 










Figure 2 Digital Reference Processes at the Department of Education 



Across most digital reference Centers, fundamental processes are the 
same: The Specialist receives a question from a customer and determines 
whether the question can be answered within the Specialist's Center. If the 
Specialist decides to answer the question, he or she may look for existing 
answers in the FAQs, archives, or Website. The Specialist then sends the 
answer to the customer. 

Under some circumstances, the Specialist refers the question out to 
another Specialist (the referent). Although many Specialists refer questions out, 
their tracking processes differ from Center to Center. The most important 
differences in the tracking processes occur at points A and B in Figure 2, above. 
At point A the Specialist decides to refer out the question to another Specialist. 
The destination for that question is decided by the Specialist. There are no 
official handbooks or guidelines about who could answer particular kinds of 
questions. Almost every interviewee had a personal list of referents; some called 
it a "cheat sheet", some a "filofax", some - just "the list". All expressed frustration 
about keeping their personal references up-to-date, and most suggested that the 
personal reference lists be shared among groups: 

it’s just a reference sheet. The Office of Legislation and Congressional 
Affairs generated this. There’s another updated version as I mentioned, 
but how updated it is, I don’t know. It’s just a subject index. There was a 
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’97 one, I believe, and there may have been a ’98. I’ve added people I’ve 
met just in the last 6 months and I’d like to someday transcribe this 
(Interviewee 2). 

and; 

A lot of times you just have to pick up the phone and say, “Hey, I’m 
sending something to you, are you the best person to send this to?” 

You have to resort to the telephone at times. There’s an incredible 
amount of knowledge in those people. A lot of grant knowledge and 
things like that... a lot of these offices don’t have a presence on the 
Web and they’re still just working with paper. Talking to them is very 
beneficial. 

Another issue emerges at point A. Currently, there are no suggested 
practices for tracking questions. Some Specialists keep tracking information in 
Microsoft Excel databases, some in spreadsheets, some on manual lists, and 
some not at all. 

A lot of the questions are the types of things that get forwarded on 
to another office and once it’s forwarded, it’s out of our hands 
(Interviewee 7). 

Tracking was frequently mentioned in the interview data: When 
interviewees were asked to talk to about their concerns (in the third part of 
the interview) many mentioned tracking. Therefore, tracking will be 
discussed in more detail in Section 3.4.2. 1 Tracking Questions. 

Point B in the figure above'idehtifies archiving as a ribther part’o^ 
referral process that is problematic for Specialists. Almost all Specialists 
remarked that it was important to store answers to commonly asked questions. 
But no two offices perform this function in the same way. Some use Microsoft 
Outlook and scan subject lines to retrieve answers. Some use more 
sophisticated databases. Some do not archive at all. This subject is discussed in 
more detail in Section 3.4. 2. 2 Archiving, below. 



3.4 INTERVIEWEE-INITIATED TOPICS 



Digital reference Centers employ impressive technological forces. They 
use routers, cables, fiber, satellites and telephony to span national borders, 
academic disciplines and languages. 

These technologies, however, are merely tools of the people who plan, 
implement and support the services. Specialists make decisions about resource 
allocations, strategic goals and acceptable outcomes - according to their own 
perspectives, or philosophies. 

Therefore, the remainder of the Results section is devoted to topics that 
were initiated by the interviewees. They may reiterate some issues discussed in 
Poll Results and Demographic Interview Results, but were of sufficient interest 
that they are presented here in detail. 




25 




3.4.1 PHILOSOPHY 



I'd like to see a message come down from the Secretary's office stating 
clearly what the Department's policy is regarding responding to email 
(Interviewee 1). 

Organizations use both formal and informal mechanisms to carry out their 
goals. Formal mechanisms (written policies, mission statements and other kinds 
of documentation) may be mandated and monitored across offices, departments 
and programs. Informal mechanisms (such as culture, norms, or philosophy) 
are not represented by tangible artifacts and may vary substantially across and 
within departments. While it may be invisible, organizational philosophy is a 
crucial component in how - and how well - Department goals are accomplished. 

Philosophy can vary by organizational function, managerial level, and 
other variables. Few organizations are able to instill the same philosophy in every 
Specialist, and this may not even be a desirable goal. It is important, however, 
that the differences in philosophy be identified and understood. To that end, the 
following observations describe philosophical conflicts that may affect the 
Department's ability to meet strategic goals. 

3.4.1 .1 WHAT IS THE GOAL? 

Some philosophical conflicts are a function of an individual's position in the 
hierarchical structure. Highly placed executives often have dramatically different 
concepts of digital reference - its purposes, features and goals - from the 
perspectives of Specialists who perform the digital reference services. One 
executive described the contrast: 



We charge fon/vard with some idea, which, in our minds, should 
work very well, but when you are down in the trenches and your 
ownership or views haven't been reflected, it makes. ..it much harder 
to overcome resistance... And they do have a little different 
perspective than I do... (Interviewee 9). 

Due to differing perspectives, then, various Specialists may understand 
differently the goals of digital reference. 



3.4.1. 2 WHO IS THE CUSTOMER? 

There is some conflict within individual digital reference Centers regarding 
the identity of the customer. Program officers, for example, may be concerned 




26 





primarily with serving the Secretary or the White House, rather than the general 
public; 



We have a system, at least from the Secretary’s perspective, that 
works pretty well. If you are a Program officer, and you are running 
a grant competition, the pressure for you is to. ..make sure the 
money gets out in time, to worthwhile projects. That’s where the 
emphasis is for these people [Program officers]. Controlling 
correspondence, e-mails, those kinds of things, tend to take on 
lesser importance. It’s just not one of the priorities for many 
Program officers (Interviewee 9). 

Digital reference Specialists who deal directly with the public have a 
different perspective about the identity of the customer: 

What's really, really, important... is that we are able to... have the 
time to spend with people to talk about how you get involved with 
improving education in your community, and the special projects and 
Programs that the public is really trying to push (Interviewee 6). 

3.4.1 .2.1 MORE CUSTOMERS 

We've been getting a lot more people e-mailing us for research. 

Now you have a lot more free e-mail accounts. So people are e- 
mailing us more. When I first started doing this, I took about 600 to 
700 a month, and now it’s about 1000 (Interviewee 8). 

Interviewees confirmed the views of poll respondents (see Section 3.2.2 
above), regarding the increase in email traffic and growing sophistication of 
digital reference questions. They report that email is supplanting phone service, 
fax and postal service and that the number of digital reference questions is rising 
weekly. As shown above in Figure 1 Human Intermediation Curve, the growth is 
a result of new customers who have new needs and various levels of technical 
and subject expertise. 

It may be useful at this point to investigate the sources of new customers, 
the types of questions they ask, and the kinds of challenges they will pose to 
digital reference Centers. 



3.4.1 .2.2 NEW SOURCES OF CUSTOMERS 

Until a few years ago, many Department offices emphasized internal 
support and regarded questions from the public as "loose questions, from 
outsiders (Interviewee 12)". Recently, however, the advent of affordable 
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technology has generated millions of questions from many kinds of new 
customers. New sources of customers were identified by Departmental 
Specialists as follows (listed alphabetically); 
citizens 
congress 

external direct request 

foreign citizens 

general public 

internal direct request 

internal referrals 

lawyers 

parents 

students 

teachers 

vendors 

the Secretary. 



This list may be only partially representative of all customer populations, 
but illustrates that digital reference Specialists and Centers must answer the 
needs of a newly diverse group. 

3.4.1 .2.3 TYPES OF QUESTIONS ASKED 

Second only to the conflict about the customer identity, is the issue 
of what questions should be answered. This issue is illustrated in the 
"Pizza Hut" story: 



Interviewee 8: 



Interviewer: 

Interviewee 8: 
Interviewer: 
Interviewee 8; 
Interviewer: 
Interviewee 8: 



Somebody wanted to know about Pizza Huts in the area, I gave 
them the Pizza Hut e-mail address, found the location site, sent 
him on to them. 

Did you consider the location of the Pizza Hut question to be 
out of scope? 

Yes. 

And, you answered it anyway. 

Yes I did. 

How come? 

Because we are customer service. 



The Pizza Hut story shows that Department Specialists possess a strong 
commitment to answering the public, and sometimes answer questions that are 
out of scope. Contributing to the problem is the fact that few Departmental 
Centers define the parameters of what is off-topic, and advise how to treat out-of- 
scope questions. Each Specialist, therefore, is left to make that decision on his or 
her own. Often, making such a policy decision is more difficult than just 
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answering the question, thus many Specialists opt to answer the out-of-scope 
questions. 

The results of many Specialists making independent policy decisions 
about out-of-scope questions may be confusing to the public, result in 
inconsistent service, and make inefficient use of resources. 

3.4.1 .2.3.1 NEW KINDS OF QUESTION CONTENT 

Each interviewee mentioned the existence of several question types and 
implied that some require special attention. The following list is an example of, 
but not the inclusive set of question types, and they are listed in the order that 
they occurred in interviews: 

on-topic-specific (can be answered with some effort using various 
resources) 

on-topic-general (can be answered easily using Center resources) 
out-of-scope (email intended for another Specialist, Center or office®) 
position clarification (requires subjective response directly representative 
of the Center's policy. This answer type is used, for example, when 
a customer asks about the advantages of private schools.) 
policy interpretation (requires high degree of synthesis and subjective 
response, "Can I sue the parents of children bullying my child?") 
technical email (intended for Webmaster 

request to update site/upload files (from Program offices and potential 
link partners) 

error messages (autoresponses such as "server down") 
maintenance (email from customer reporting server difficulties) 
listservs (customers encountering difficulty signing up for/posting to 
listservs). 

Each type of question presents unique challenges to the Specialist and 
the Center, and should be addressed in training. Out-of-scope questions present 
an especially large potential risk; that of wasting resources, and is discussed in 
greater detail below. 

3.4.1. 2.3.3 OUT-OF-SCOPE QUESTIONS 

No matter what the intended scope for a digital reference Center, 
Specialists often receive questions that are unrelated to their Programs and 
Centers. These are out-of-scope questions, and interviewees had several 
definitions, listed here: 



® "The new Internet service at vote.com asked visitor to click on Yes/No radio buttons on its home 
page to vote on school vouchers. Each Yes/No vote triggered a form email to customer service 
@inet.ed.gov. Poor (Specialist's name) received a thousand such emails in a week. ...It's not out- 
of-scope but it's certainly unwanted" (an example of out-of scope email from Keith Stubbs, 
December 18, 1999). 
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unnecessary and repetitive follow-up dialogues ("Customer isn't listening") 

mistakenly sent messages (sent by customers) 

inappropriate referrals (sent to wrong Specialist by other Specialists) 

pornographic messages (from unknown individuals) 

advertisements (from unknown individuals) 

unsolicited overtures (from vendors already serving the office) 

grassroots persuasion messages (requests for support political) 

crank mail (“Close NASA now") 

hate mail (from unidentified individuals) 

death threats (from unidentified individuals) 

security threats (from unidentified individuals) 

viruses (from unsuspecting customers) 

requests for links to other sites (from other organizations) 

dead-link information (about links for which ED.gov is not responsible) 

upgraded pages (should be updated on customer's server, not ED. gov's) 

error messages (redirected email from customers re: jammed servers). 

While interviewees have a clear and collective knowledge of out-of-scope 
questions, many are unclear about what to do about them. Similar in its negative 
outcome is the conflict over what comprises a good answer. 

3.4.1 .2.4 WHAT IS AN APPROPRIATE ANSWER? 

One interviewee pointed out that there is little agreement about how to 
answer a question. Answers depend on how committed each individual manager 
is, and some perceive that their managers are not as committed to customer 
service as the Specialists are; 

I think our boss looks at it as a pain-in-the-butt job that we have to 
do. He told me to answer things as briefly as possible and just keep 
it short and sweet and don’t be really overly friendly to people. He 
just wants us to forward it (refer it out to another Specialist). He just 
wants us to jettison it out of there (Interviewee 1). 

Keeping it "short and sweet" may actually be beneficial to both the 
Department and to customers, who may not want to wade through several 
pages of information. And coordinating questions with external Specialists 
may, in fact, provide better answers to customers. But as long as these 
decisions are perceived as mere expediencies. Specialists may not agree - 
with each other or with managers - regarding how to answer questions. 

3.4.1 .2.4.1 TYPES OF ANSWERS 

Each Center administrator prescribes - either formally or informally - the 
format and nature of an acceptable answer. Thus, the contents of answers vary 
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by Center - and often by the kind of customer requesting information. Acceptable 
answers may contain one or more of the following kinds of responses (listed in 
order of increasing complexity): 

citation (name of a resource material) 

pointer (name of resource material and instructions for accessing it) 
full text (text from a resource material) 

statistic (data with minimal context, usually numeric, brief answers) 
referral (notice that the question has been sent to another Specialist) 
research (list of citations from ready reference materials) 
customized research (citations/pointers resulting from detailed search) 
synthesis (any of the above with explanatory verbiage) 
compound (any combination of the above responses). 

These differentiations among answering techniques are only implicitly 
understood in the Centers. If identified and strategically deployed, the techniques 
could be used to streamline and standardize processes. 

3.4.1 .2.4.2 SOURCES OF ANSWERS 

Too many sources! We normally keep 5-6 things open on our 
desktops - sometimes Netscape and Internet explorer, and all my 
different bookmarks, clearinghouses, and the databases - all the 
different things open. If there was a way to have all the information 
accurate and use one depository where we wouldn't have to keep 8 
or 9 things open - would be good (Interviewee 3). 

As the number of online organizations and Centers grows, sources, 
references, and links to other sites, grow exponentially. This situation 
slows down the Specialists' response times in two ways. First, each 
source must be evaluated for legitimacy and currency of information. 

Second, each source has a different interface and different navigational 
standards. 

The growing number of resources is oven/vhelming Specialists with 
information, new interfaces and greater expectations for service. 



3.4.1.3 SUMMARY 

Philosophic differences result in conflict and wasted resources across the 
Department's digital reference Centers. These differences are sometimes linked 
to an individual's position, and often create confusion that can be expressed as 
the following questions: 

What is the goal of this digital reference Center? 

Who is the primary customer? 
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What questions should be answered? 

What are the parameters for determining out-of-scope? 

How should we treat out-of-scope questions? 

What are the characteristics of a "good" answer? 

Knowledge management and human resources are affected by these 
philosophic differences. Those issues are explicated in the following sections. 

3.4.3 KNOWLEDGE MANAGEMENT: TRACKING, ARCHIVING AND 
FAQS ACROSS PLATFORMS AND MEDIA 

We have created a system of information that is going out to the 
public, (but) we didn’t take it a step further, which is, - now what do 
we do with it, once it comes back to us? (Interviewee 9). 

Before they respond to a digital reference question, most Specialists 
record data about the question. This information helps Specialists monitor the 
progress and status of each question, and is called Tracking. 

When a question is answered, the response is often stored and indexed 
so the answer may be re-used. This is known as Archiving, and successfully 
implemented, can reduce costs and increase efficiency of digital reference 
Centers. Tracking and archiving at the Department's Centers are briefly 
described below. Following that is the presentation of a number of challenges to 
effective tracking at the Department's Centers. 

3.4.2.1 TRACKING QUESTIONS 

It would be really nice, if there was some way to track these 
questions ...if we got a receipt, and knew that (referents) .. .actually 
opened the mail and are answering it (Interviewee 8). 

Tracking has two fundamental uses: it allows Specialists to track the 
status of a question and report that back to the customer, if needed. Equally 
important, tracking data can be analyzed to find strengths and weaknesses in 
customer service. Tracking statistics allow evaluation of turn-around time and 
resource allocation. These functions are components of knowledge management 
and their successful implementation is crucial to effective digital reference 
Centers. Currently, however, tracking at the Centers is hampered by lack of 
consistency, as described below. 

3.4.2.1 .1 TRACKING LACKS CONSISTENCY ACROSS REFERENTS 

Although standards do not dictate what must be recorded across Centers, 
the following fields are most commonly recorded: 

date (of receipt of the question) 
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name (of customer) 
email address 

mode of contact (Web page, email, other) 
kind of customer (student, teacher, parent) 
due date (according to policy) 
topic 

name of Specialist 

referent (was the question fon/varded on to another Specialist?) 

The last field (Referent) is key to tracking activities: The second Specialist 
(the referent) may have personal expertise with which to answer the question, or 
an appropriate FAQ. When Specialists send questions on to referents they often 
track the progress and status of the question. 

Regardless of which Specialist finds the answer, the customer's 
perception is that the first Specialist, rather than the "invisible" referent, is 
answering the question. Many Specialists, therefore, expect referents to return 
the answers to them (so they may send answers back to customers, along with 
descriptions of how they obtained the answers). These Specialists often want to 
track the progress or status of the question (e.g., "received by the referent", 
"being researched", or "being returned with answer"). Some of these Specialists 
also want to rate the answers for quality and speed and be able to report these 
data back to their own managers and the referents' managers. 

Other Specialists, however, offload to referents not only the questions, but 
the responsibility to send the answers to the customers. These Specialists either 
want a brief electronic receipt of the question, or do not require tracking of further 
actions. 

Those are the types of things that get fonwarded on to another office 
and once it’s fon/varded, it’s out of our hands (Interviewee 5). 

Because tracking procedures are not standardized, each office - and often 
each employee - uses different tracking systems: 

Each office and department has got their own internal system for how they 
route and control (Interviewee 9). 
and 

We put a lot of stuff (tracking data) into Lotus notes. Let me 
rephrase that - 1 put a lot of stuff into Lotus notes (Interviewee 8). 

Some digital reference Centers are operated by one individual so 
idiosyncratic methods of storing and tracking data are not a problem: 

as it's fon/varded on, it’s actually just saved in a folder and the folder 
is simply called "Chris" (Interviewee 4). 
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Most Centers, however, are operated by several Specialists, so 
individually stored tracking systems and inconsistent file-naming conventions 
present barriers to easy tracking. 

3.4.2.1.2 TRACKING LACKS CONSISTENCY OVER TIME 

There is little continuity in tracking requirements over time: 

When I assumed this responsibility our project manager told me not 
to worry about tracking things after I forward them... I was a little 
uncomfortable with that and I asked him why. He said, “You don’t 
need to be spending too much time on this." But (my predecessor) 
took such good care of it and she’d always say (to customers), " If 
you don’t hear from so and so contact me in 2 weeks” (Interviewee 
1 ). 

3.4.2.1.3 TRACKING: CONSISTENCY ACROSS TOOLS 

In some offices, there is little effort to match tracking tools to whatever 
tracking requirements do exist: 

We were supposed to be reporting back... on a monthly basis. It's 
hard to try to keep track of that stuff... It's a nightmare. It's extremely 
tough and time consuming to try to jam things into (electronic) mail 
boxes, saying "here's this subject"... (Interviewee 1). 



Some Centers use manual tracking systems. Others use Microsoft 
Access, Outlook or LotusNotes. Still others use 'home-grown" systems based on 
customized software programs. 

Many Specialists report that they are researching new software, but there 
is no organized effort: working in isolation, many are looking at the same 
software packages, and have no mechanism for comparing findings, and making 
decisions about software selection or creation. 



3.4.2.1 .4 TRACKING: CONSISTENCY ACROSS MEDIA 

Further fragmentation exists in the way that multi-media messages are 
tracked. No Center participating in this study has accomplished the tracking of 
phone, fax, paper and email digital reference questions. Thus, answers are re- 
created many times across kinds of media (see section 3.4. 2.4 Platform and 
Media, below). 

3.4.2.1.8 TRACKING: CONSISTENCY ACROSS VARIOUS LEVELS OF 
COMMITMENT 



ERIC 
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Also fragmenting the Centers' efficiency are the various philosophies 
about the importance of tracking. Some managers view tracking as an 
administrative task only, and consider it to be unrelated to, or not useful for 
improving customer service: 

To the degree that you are putting resources into tracking and 
monitoring, you are getting away from what the purpose of the job 
is... (respondent 9). 

Tracking was viewed most negatively by the respondents holding the 
highest positions in the Department hierarchy, each of whom has many 
responsibilities aside from digital reference: 

For many program people - responding to letters is not one of their 
favorite things to do. It's just not one of their priorities for many 
program officers. I want to know what the purpose of what we’re 
attempting to achieve and what are the trade-offs in achieving that 
(Interviewee 9)? 

Specialists who were devoted entirely to digital reference responsibilities 
were the most enthusiastic about automating, standardizing and otherwise 
improving tracking procedures. 

3.4.2.1 .6 TRACKING: CONSISTENCY ACROSS COGNITIVE STYLES OF 
INDIVIDUALS 

Because the Centers have grown up quickly, and somewhat separately 
from each other, the daily routine of tracking questions has evolved from the 
habits and cognitive styles of the Specialists, not planners or administrators. For 
example, the frequency with which Specialists check their email depends upon 
personal decisions, not policy. Some Specialists check their emailboxes five 
times daily. Others read email once a week. 

Similarly, prioritizing is an undocumented process, and differs from 
Specialist to Specialist. Some Specialists prioritize by subject line, some by 
sender, some by the order in which the messages were received. One 
interviewee scans each message to see whether the word "urgent" is in the text. 

3.4.2.1.7 TRACKING: CONSISTENCY ACROSS MARKETING STRATEGIES 

While Specialists are generally very committed to answering, their 
enthusiasm for serving the customer may exceed their resources: 
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Generally, I contact (the customer), and give them the program 
name, my number and my personal phone number, if they have 
question, to give me a call (Interviewee 10). 

This constraint becomes a major impediment as new Centers emerge: For 
example. Interviewee 10 reported that her digital reference Center is new and 
very popular. Her office has delayed publicizing the Center until processes are 
optimized. What will happen when the site is publicized? When students, 
parents, teachers, researchers and journalists are cutting, pasting and informally 
disseminating information about the Center on listservs, Web sites and in school 
newspapers? It is unlikely that this Specialist will be able to provide the level of 
personal service she describes above. Currently, there are no policies that 
recommend minimum levels of service, nor are there recommended processes 
for managing spikes in volume or for offloading questions to other Centers. 

This is an important point for the Department because use of the Centers 
is one way that the public forms opinions about government entities. 

When someone sends an email message to the Website of a 
government agency, that is often the first and only contact they will 
have with that agency. (Interviewee 4). 

In summary, tracking allows digital reference Specialists to monitor 
question status and provide information about the quality of customer service. 
Tracking is especially important to Specialists who refer questions to other 
Centers. Most Specialists track the same kinds of information, but use different 
tools to perform tracking. The result is inconsistency in tracking across; 
referents (other Specialists) 
time 
tools 
media 

commitment levels 

idiosyncrasies of individual cognitive styles, and 

marketing plans that are not aligned to the Centers' resources and goals. 

Fragmented systems and inadequate policy hinder the ability to share 
tracking information and solutions across Centers and media. 

3.4.2.2 ARCHIVING 



Everyone has their own way of doing things. Everyone has their own 
mail archives (Interviewee 1). 

Archiving is the methodical storage of answers to digital reference 
questions. Archived questions and answers can be stored in electronic files 
and retrieved and re-used to answer repetitive questions. They may also 
be used to construct online information resources called Frequently Asked 
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Question Centers (see the Section 3.4.2. 3 FAQs). It may be helpful to 
review the relationships among tracking, archiving and FAQs. 

Tracking is the monitoring of a question's progress and status until 
the answer is sent to the customer. 

Archiving is the storage of the questions' answers either in 
databases, online sites, text files or in hard copy 

FAQs are teaching tools presented in the format of questions and 
answers. Their content is derived from archived answers, and is 
accessible to customers through a Web site interface. A FAQ may be a 
definitive collection of clarifying questions and answers on a topic, 

(e.g. http://www.shu.edu/aboutAAAAAA/Faq), as a general collection of 
anticipated questions and answers (e.g., http://www.ed.gov/faqs.html), or 
as a generic collection of questions and answers. 

Like tracking, archiving is a fragmented process that differs from 
Center to Center. 

Documents on my laptop and PC, are mostly canned responses, or 
sources of information that I have to constantly update. If 
somebody wants the State Department of Education- for some 
reason these guys like to change their URL's’ a lot, so I have to 
constantly get through that. But I keep a list. It just makes it a lot 
easier, but the problem is I do have to constantly go through, 
because the states will change their site. It’s (filed) under My 
Document. I know what the names mean, just from sheer 
recognition (Interviewee 8). 

Currently, archiving processes are diverse and informally managed. 
Because no formal policies guide archiving, any impediment can halt the 
process. Inadequate software, for example, can limit the ability of 
Specialists to archive data: 



Pine (a popular email software application) slows down if you keep 
too much archived in it. So these records (answers to frequently- 
asked questions) are kept only two months. They got rid of earlier 
ones. Also with Pine, only one person at a time can work with the 
records (Interviewee 1). 

and: 

I did it (archiving) for about six months or so, until the night 
that Outlook (a software application) mail blew up. It just blew 
up (Interviewee 1). 
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Archived answers are used to share knowledge across departments and to 
supply consistent and convenient responses to frequently asked questions 
(FAQs). 

An additional concern - one often overlooked by busy managers - is 
preserving the accurate history of information about Department programs, 
grants and documents. Unlike paper resources, online sources are constantly 
changing, and the Department cannot depend on libraries to safeguard historical 
material. Legal and medical librarians, for example, are trained to preserve 
accurate histories of case law, and medical procedures. This information is useful 
in litigation, education and describing historical context. As of this report, the 
Department has not made clear its policy regarding the preservation of digital 
reference answers. 

3.4.2.3. FAQS 



It's harder to find them (FAQ's) than it is to just make up answers 
(Interviewee 13). 

Most digital reference Centers use Frequently Asked Question (FAQ) files 
to store answers to commonly-asked questions. As mentioned above, FAQs are 
educational tools that are developed from archived answers to previously asked 
questions. Entire FAQ files may be devoted to popular topics, and they contain 
complete, thorough and accurate information that has been captured and stored 
in the archives. These files are located on the computers of digital reference 
specialists, and can be scanned and retrieved for use in answering frequently- 
asked questions. 

Some Specialists create their own FAQ collections and use them 
consistently; 

...you can capture the information so that it can be re-used, so that 
you can make the next response faster (Interviewee 9). 

For other Specialists, however, finding those answers can be sufficiently 
difficult, that many avoid searching the FAQs and generate answers from their 
own experience: 

I find it interesting that it’s not, "Oh, let me go look up the answers 
that somebody put together" (Interviewee 4). 

Finally, some Specialists are concerned about the information quality of 
FAQ resources: 

A lot of people get this stuff off the Web, and it’s so out of date that 
it’s scary. It horrifies me because people think - 1 mean this is the 
classic thing - even people in my office think, well this is on the Web, 
so that’s it! 
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And, I say, "Shouldn’t we call these people, to make sure the 
information is right? Don’t just depend on one source of information." 

We probably give different answers than what the National Library 
gives, or what ERIC gives, or the Webmaster is giving, or whoever. 

I don’t know if there is any coordination (Interviewee 3). 

In summary, there are many reasons that digital reference Specialists may 
avoid using FAQ files, including: 

There is no consistency to naming and storing FAQ files and they are 
difficult to find. 

The files were created by someone else with a different way of organizing 
and storing the files. 

There are so many FAQs that searching them takes longer than 
generating an answer from scratch. 

Some FAQ information may be obsolete. 

Some FAQ information may be inaccurate. 

FAQ information may be inconsistent with other official resources. 

FAQs represent a powerful tool with which to answer rapidly 
increasing emailed questions. But navigation, the capacities of human 
beings to remember what is stored where, the currency of FAQ information, 
and the flagging ability of searching mechanisms, are all impeding FAQ 
use. 

3.4.2.4 PLATFORMS AND MEDIA 



These different pockets have developed (in various offices). They have 
developed... their own everything, so there is a lack of coherence... So, I 
think we have to look at what’s grown up, and see what makes sense in a 
distributed organization (Interviewee 9). 



Department of Education digital reference Centers support the needs of all 
customers, regardless of their communication modes or technological platforms. 
Digital reference answers are generally sent using the medium in which the 
question was received, whether by mail, phone, fax and email. Each medium has 
generated the creation of separate tracking systems for each digital reference 
Center. As a result, phoned questions and their answers are tracked differently 
from mailed and emailed questions. In some Centers mail, phone and email 
questions are tracked by different personnel, in different offices using different 
tools. Therefore, answers are not shared across communication modes, and 
must be continually recreated. Respondents to the poll and the interviews predict 
that the percentage of telephone and mail questions will taper off as email traffic 
grows. The shift to email messages alleviates two problems: Phone logs show a 
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high abandonment rate: that is, phone many customers hang up before their calls 
are answered. 

The second problem, the inability to capture answers, is inherent to 
telephone communication: 

One of the things that you can’t do on the phone, but that you can 
do on email, is that you can capture it (the answer). So, in a phone 
call, if someone comes up with the greatest, most brilliant, most 
wonderful response, (and then) they walk out the next day and get 
hit by a bus - there goes that knowledge. With e-mail you can 
capture that information so that it can be re-used, so that you can 
make the next input faster in response. ... 

Despite the convenience of digital reference, some customers do not 
have access to email. That circumstance prevents (at least for now) a mandate 
to move all reference activities to email. Even if such a mandate existed, it would 
not solve the problem of "technology islands" that prevent sharing of knowledge 
bases across Centers and programs: 

Presently because of the lack of interconnection with systems, 
there is a recreating of that message every time an email goes out, 
and in whose head it is? And how do we make sure that we can get 
to that person (Interviewee 9)? 

Each Center currently selects its own database platform, processes, 
operating system and applications. Software choices across Centers include, but 
are not limited to; manual systems. Pine, Microsoft Outlook and Access, Lotus 
Notes, Unix-based (such as iNet, Pine), custom applications (e.g.. 
Correspondence Control Manager or CCM), and Teloquent for phone tracking. 
Firewalls create barriers between many Specialists. Web browsers are rapidly 
evolving and new upgrades often prevent customers from filling out Web-based 
forms, and using the email connections on Department Websites. Systems 
personnel are facing the increasingly difficult task of allowing their systems to 
communicate with each other. 

In summary, tracking, archiving and FAQs - important functions of 
knowledge management in digital reference Centers - are impaired by diverse 
platforms, media and procedures. 

3.4.3 EXTERNAL INFLUENCES 

The biggest problem we have is that some journalist or somebody 
up there will write this (a new Program service) up and not tell us 
(Interviewee 8). 

Outside of ED.gov, many digital reference Centers can anticipate and plan 
for external influences. They possess mechanisms for tracking developments 
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and they update their knowledge bases as new developments occur. Few of 
them, however, face the kind of external influences that challenge the 
Department's Centers. 

For example, while a medical digital reference service may routinely 
monitor for new developments in telemedicine, research and clinical practice, 
new developments are rarely of a magnitude to create confusion, delay service 
or create new policy. In the Department of Education's digital Centers, however, 
sudden, externally-initiated developments may lead to surging email traffic, 
oven/vhelmed staff, and confusion about how and what to answer. These 
developments are generated by two kinds of external influence: current events 
and journalistic activity. 

3.4.3.1 CURRENT EVENTS 

Current events, for the purpose of this discussion, are the sudden, 
random, emergence of government-related events that stimulate great interest 
from the public. They include but are not confined to new legislation, international 
and domestic incidents and disasters. Interviewees pointed out that whereas 
expertise or domain of a Center affects the number of topics about which 
customer ask questions, the actual volume of questions is influenced by the 
magnitude of current events: 

Traffic tends to peak and valley depending upon what issues or 
controversies might be relevant to that particular time (Interviewee 9). 

Current events present a particularly difficult challenge because there is a 
chronological gap between the event itself, and the government's official 
response. During that brief time, digital reference Specialists - many of whom are 
committed to 24 hour turn-around service - are unprepared to answer topic- 
related questions. Compounding the confusion is the sudden increase in 
incoming email questions: 

Email jumped up tremendously especially when we had the school 
shootings, and I have to admit the last time with Columbine, I don’t 
know what happened (Interviewee 8). 

Interviewees reported that the Columbine incident affected all of the 
Centers. Some interviewees reported that press releases that went out to the 
public, did not go to the Centers. Others expressed concern that incorrect 
information (about safe schools) was disseminated, and still others spoke of the 
diminished quality of service resulting from message overload and longer waiting 
times for answers. The Columbine incident was addressed on an ad-hoc basis 
that seemed to work well and could provide insights into planning for future 
current event traffic spikes: 
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They got a Website up within a couple of days... they did a really 
good job of it, of getting a Website up and a specific e-mail address. 

We had a Website to send them to, and an e-mail address to send 
their comments, and that took just a couple of days, and it was 
great. It made it a lot easier (Interviewee 8). 
and 

After Columbine, I had loads of emails. We waited a day or two (to 
respond). And then we sent out an Edinfo, reminding people that a 
guide was designed to help. We put it on our home page. And then, 
the President told people in a town meeting. He even published our 
email address (Interviewee 13). 

The anecdotes above suggest that while individual current events cannot 
be anticipated, some steps should be taken to avoid; 

(a) dissemination of incorrect or inconsistent answers 

(b) confusion in the Centers and 

(c) diminished service due to longer turnaround times. 



3.4.3.2 JOURNALISTIC ACTIVITY 

When significant current events occur, journalists write about them, and 
their publications often affect the operations of the Department's digital reference 
Centers. For the purposes of this discussion, journalistic activity is defined in two 
ways. The first definition of journalistic activity - publications by professional 
journalists - is easily understood. Professional journalists often publish material 
that refers to the Department's digital reference Centers. But they do not alert the 
Department to anticipate the inevitable increases in demand. 

Sometimes confusion results because customers refer to the newspaper 
they read, instead of the Departmental Program or information that was the 
subject of the newspaper article: 

When a newspaper writes up the (Department) publication, it would 
be darn nice if they would put the name of the (Department) 
publication in there. Let's say they come up with a new reading 
report, and it’s in the Post and in USA Today, but they don’t always 
tell you what the name of the report is (Interviewee 8). 



That’s one of the bigger problems - people will see this (a 
Departmental article) referred to in their local newspaper, and they’ll 
email us. We have a publication called Helping Your Child Learn 
Science . One issue featured an experiment with celery entitled “The 
Celery Stalked at Midnight”. We keep getting these requests for 
“The Celery Stalked at Midnight". It’s like, what the heck are they 
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talking about? We had no clue and it adds to digging (Interviewee 

8 ). 

When customers send reference questions to the Department's Centers, 
they often refer only to an article's title as mentioned in a journal or newspaper. 
They don't include the name of the Departmental program or service in which 
they are interested. This increases the workload for digital reference Specialists 
who must then obtain a copy of the publication to look up the reference. 

A second definition of journalistic activity is the cutting, pasting and 
electronic dissemination of previously published articles by journal readers. This 
secondary journalistic activity - until recently almost invisible - has emerged as a 
powerful influence on digital reference Centers in the Department. The fact is that 
any teacher, parent or student can cut and paste press releases and online 
articles to education-based-listservs and individuals all over the world. And they 
are free to do so without asking permission, registering the activity or othen/vise 
giving notice to the Department. This phenomenon is an example of the Web's 
powerful capacity for education. It should not (and logistically cannot) be 
curtailed. It does, however, create unexpected demand for services and must be 
addressed to relieve duplication of effort. 

Sometimes it (email volume) just goes up, for no apparent reason. I 
received 20 e-mails in the course of 3 days, saying, “Please send 
me 50 copies of the Parents Guide to the Internet And I have no 
idea where it came from. 

Ed Info stuff gets cut and pasted and put all over everywhere. 

You can find it in surprising places, tucked away here and there, on 
the Net, as well as in state newsletters, and various education 
newsletters (Interviewee 13). 



Center managers may be tempted to discourage informal dissemination of 
information that will stimulate heavy use of the Department's digital reference 
Centers. But the success of a digital reference Center is often measured by the 
frequency of its use. Thus, the burgeoning interest that is generated by informal 
dissemination of Department information must be accommodated rather than 
curtailed. 

In summary, the Department's Centers face two kinds of external 
influence, current events and journalistic activity. Current events occasionally 
cause traffic spikes, dissemination of inconsistent answers, and diminished 
service due to longer turnaround times. 

Journalistic activity - both professional reporting, and informal 
dissemination of articles - is becoming a challenge to Center policies, processes 
and goals. 

3.4.6. HUMAN RESOURCE ISSUES 



O 

ERIC 
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Digital reference Centers are dependent upon the abilities of human 
beings to answer customers' questions. As more people have access to 
networked resources, customer populations grow and change, placing increasing 
pressure on Specialists who are unable to change their cognitive limitations. 
Specifically, Specialist must find ways to manage: 
more customers 
new sources of customers 

new types of questions, including out-of-scope questions, and 
new types of answers (various formats and levels of sophistication). 

The digital reference Center function is not always understood or 
appreciated by organizational managers and administrators. Some use the 
function as a temporary job for employees they cannot othen/vise employ. In such 
cases, digital reference is served by under-skilled workers who may not know or 
care about the Center's mission: 

A lot of these jobs go to people who need something to do. Don’t just 
shove it onto someone because we need to give them something to keep 
their grade, or get them out of my hair, or to stop them from twiddling their 
thumbs all day (Interviewee 8). 

Organizational pressures such as employee placement and lack of 
resources may leave some Specialists ill-equipped to acquire new skills. This is 
profoundly detrimental to the Department and its strategic plans for digital 
reference, and is explored in the following section. 



3.4.5 NEW SKILLS 



Preceding sections have reported some developments that affect 
interviewee's skill sets. This section is intended to list all such new skills as 
mentioned by interviewees: 

4.4.5.1 DIFFERENTIATE SERVICES 

Specialists must be able to recognize and work with customers who have 
varying technical skills and levels of sophistication about topics and resources. 



4.4.S.2 UNDERSTAND POLICY AND STANDARDS 



"Trying to explain why we have this but not that (Interviewee 2)" can be 
difficult for Specialists in large and complex organizations. They must understand 
their organizations' policies, and the standards for delivering information. 

4.4.5.S EDUCATE CUSTOMERS 

...with people trying to use these resources in their offices or 
homes, we're having to do (customer) education by phone ore-mail 
(Interviewee 7). 

Specialists often have to decide whether to instruct customers in how to 
find material - or just provide it themselves ("...we tried to explain, 'No, you can't 
download to your television set'," Interviewee 1). 

Customers often don't know, "... how much more they need to KNOW 
before they can even begin to use these new resources well (Interviewee 3)". 
Specialists must be able to (a) make the decision when to instruct and (b) have 
the resources and skills to provide appropriate training. 

4.4.5.4 LEARN NEW SOFTWARE 

Effective management and operation of digital reference Centers is 
dependent upon the ability of Specialists to use their tools. For digital reference 
Specialists, the tools include multiple databases, communications applications, 
presentation and email applications. Additional tools include reporting, telephony, 
contact management and record management applications. 



4.4.5.5 EXERCISE MANAGEMENT SKILLS 

As Specialists exchange resource lists and augment their reference 
resources, they will need to practice contact management. Good communication 
and record management skills will contribute to the accurate and timely 
management of their contacts. 

4.4.5.6 MONITOR AND UPDATE KNOWLEDGE MANAGEMENT AND 
RECORD KEEPING SYSTEMS 

Specialists reported that they are unaware of processes for updating 
archives, FAQs and other knowledge management systems. 



4.4.5.7 PREPARE FOR FLEXIBLE, SHIFTING RESPONSIBILITIES 



Because Specialists are shouldering more tasks, each one must be able 
and prepared to do many kinds of tasks. 

4.4.S.8 LEARN AND MAINTAIN GOOD WRITING SKILLS 

We want to develop shared standards of quality in writing, and what a good 
response is (Interviewee 13) 



and 

Some people have terrible netiquette. I sent one over to (another Specialist) 
in non-public schools office, and he actually corrected the grammar of the 
(initial Specialist) that sent the e-mail. 

Because many digital reference Centers re-use answers in Websites, 
archives and FAQs, Specialists must view their writing and communication skills 
as permanent and strategic additions to the Department's core capabilities. 

The skills listed above were mentioned by interviewees as issues that they 
face in operating digital reference Centers. The stories that the interviewees told 
suggest that another list could be constructed - a list of required skills for Center 
administrators and managers. Since they are implications of the data, and not 
explicit excerpts, those suggestions will be explored in the Section 4.5 Training 
Managers. 

In summary, specialists must contend with new challenges, including, 
more customers, new sources of customers, out-of-scope and new kinds of 
questions, and new types of answers. Resource and organizational limitations 
are not supporting Specialists in acquiring newly necessary skills such as: 

• differentiating services 

• understanding policy and standards 

• educating customers 

• learning new software 

• exercising managerial skills 

• monitoring and update knowledge management systems 

• preparing for flexible, shifting responsibilities 

• learning and maintain good writing skills. 
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4 ISSUES 



After data collection, the research team reexamined the poll respondents' 
and interviewees' observations (presented in Section 3 Results, above), and 
grouped them in issues. The issues are listed in the chronological order that is 
most common in strategic planning, and includes: 

Sponsorship of digital reference 
Policy refinement 

Standardization of processes to establish policy in processes 
Software selection and or design 
Training to maintain policy in operations 
Resource sharing. 



4.1 SPONSORSHIP 



Section 3.1 shows that the Department supports excellence in digital 
reference and section 3.4.1 suggests that the Department's philosophy 
understood by all Specialists. But the high-level philosophies have not been 
converted into actionable plans and there are few documented standards or 
prescribed procedures for the use of email-based digital reference. 

Coordination of processes is occurring on an ad-hoc basis, but is left to 
individual managers in each Center. This ensures that the needs of Specialists 
will be identified, but does not ensure that the necessary resources - usually 
supplied by a high-level champion - will be provided. A highly placed individual, 
committed to and knowledgeable about digital reference should be identified. 
This individual must be recruited to sponsor, coordinate, procure funds for, and 
coach the process of re-inventing digital reference Centers at the Department. 



4.2 POLICY REFINEMENT 



Policy supports an organization's goals by articulating explicitly how to 
interpret them into standards. Standards are implemented in training, daily 
operations, and evaluation activities. Current policy insufficiently addresses the 
Centers' standards for clarification of, and centralization of processes for: 
response time 

question overload and to whom questions should be referred 

how to deal with out-of-scope questions 

scope of Centers' knowledge domains 

how they should be tracked, and 

how to design and share FAQs and archives. 

In the interests of space, we do not list every issue here, as each is 
discussed in greater detail in the following section. Standardization. 
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4.3 STANDARDIZATION 



Standardization is the process of making procedures consistent across 
populations, and is often viewed with trepidation by those it affects. Some view 
standardization as the eradicator of individual creativity. Properly implemented, 
however, standardization can reduce friction (repetitive, non-productive or 
frustrating efforts that detract from productivity). More importantly, it can free 
Specialists to use their time and resources to act as topical experts, rather than 
record-keepers. 

Virtually every complaint voiced by participants in this research is related 
to the fact that processes are fragmented, confusing, and undocumented - in a 
word, unstandardized. It is well to remember, however, the speed with which the 
World Wide Web has been adopted by the Department, and to understand that 
processes evolved as Specialists "put out fires". The collective expertise and 
ingenuity in putting out fires has resulted in many informal standards. Rather than 
impose them across Centers, these informal standards should be used as a 
springboard for creation of centralized standards that will coordinate the 
Departments' Centers. That task should be undertaken by the champion or 
sponsor of digital reference Centers. 

Properly developed standards can incorporate the benefits of 
centralization (streamlining, processes, preserving resources, ensuring constant 
quality) with the benefits of decentralization (better fit of processes to 
organization, customization of services for customers). To meld both 
perspectives, standardization must consist of two parts; determining the general 
degree of centralization, and determining which individual tasks and processes 
should be standardized. See Section 5.2 Centralization for a more detailed 
discussion of this process. 

For recommendations regarding standardization and centralization of 
processes, see Section 5.2 Determining the Level of Centralization. 




4.4 SOFTWARE 






Currently, ED.gov Centers use various software packages to track 
questions and answers, and individual Centers have begun research into 
software packages and systems that could: 

use shorter URLs (some are too long to paste into Web forms) 
resolve of case sensitivity issues^ 

capture URLs on email messages sent from customers using Web forms 
automate tracking and archiving 
automate checking for dead links® 

provide a capability for local private entries as well as shared, ED-wide 
entries and global public ones (at least 3 tiers of private/public) to 
provide enough incentive for distributed update responsibility. 

Section 3. 4. 2.4. showed that Centers do not share database platforms, 
processes, operating system or applications, and that because they use different 
packages, their results cannot be compared across digital reference Centers. 

A decentralized approach for selecting software will result in more diverse 
systems, and increase the existing difficulty in sharing data and communicating 
across Centers. 

These two trends - the growing diversity of software systems, and an 
increase in customer demand, require that the Centers and Specialists 
accomplish scalability - the ability to replicate human expertise through use of 
technology (Section 3.2.2). Scalability is crucial to the Centers for this reason: 
The success of information services is often evaluated by their use; the more 
questions the more successful the Center. Yet answering digital reference 
questions is labor intensive, and for that reason, many Centers are not eager to 
publicize their services. This cycle keeps many services small, and prevents the 
marketing that will procure continued funding. 

...we’ve just been seeing what would happen if we set it up and 
how would we handle this, before we publicize to the world that 
these kind of services are available online (Interviewee 3). 



Another issue for Specialists is tracking multi-media messages. 
Answers are generally sent to customers using the medium in which the 
question was received whether by mail, phone, fax or email. In some 



^ "Case sensitivity is a non-optional characteristic of UNIX web servers and cannot be eradicated 
without moving the ED web sites to Windows NT, which would have massive implications for 
capacity and support" as per email from Keith Stubbs. The research team recommends rather 
than eradicating case sensitivity, finding a software-based workaround, perhaps based on a look- 
up table for lower and upper case letters. 

® Note; Regarding software requirements for this item, automated dead link checking for email 
systems requires more sophisticated software than pinging Web sites. Diagnostic messages also 
take more time to be delivered for email than for Web-site messages. The Department is currently 
investigating ways to validate Exchange email addresses and should have a quarterly validation 
routine in place by February, 2000 (email from Keith Stubbs, December 18, 1999). 
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Centers mail, phone and email questions are tracked by different 
personnel, using different tools. Answers are not shared across 
communication modes and must be continually recreated. 

Some software packages feature the ability to track multiple modes 
of communication (phone and email, for example) and: 

The end game will surely go to leading providers of customer 
relationship management solutions that handle a combination of 
voice, email, fax, and internet transactions as easily as voice calls 
are handled today. ("When the Internet meets Call Center", 
lnter@ctive Week , page c-2. May 10, 1999, Frank Bowers). 

Recommendations for accomplishing scalability through software are 
presented in Section 5.5. 



4.5 TRAINING SPECIALISTS AND MANAGERS 



Training in the Departments' digital reference services is minimal 
and inconsistent. Only those at the "front lines" (Specialists) receive any 
kind of training and that training may or may not address such issues as: 
differentiating among types of questions, customers and answers 
understanding policy, standards and procedures, including tracking, 
archiving and referring questions 
educating customers of varying expertise 
learning new software 
exercising managerial skills 

monitoring and update knowledge management systems 
preparing for flexible, shifting responsibilities 
learning and maintain good writing skills 
responding to security threats 

Poll respondents and interviewees also raised the issue of training 
managers, and suggested the following skill be taught to them: 

creating new staffing models to select and train staff who can rotate 
responsibilities 

planning backups (cultivate Specialists domain expertise and share 
expertise across services) 

fighting for resources, especially telephony support and training 
dealing with international support issues 

directing records management and ensuring consistency across Centers 
ensuring redundancy (preserve government digital documents that may 
othen/vise disappear) 
marketing and publicity. 



4.6 RESOURCE SHARING 
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Section 3.4.2. 1 states that fragmented approaches in tool design and 
tracking systems make it difficult to share tracking information and solutions 
across Centers. Specialists commented on the lack of ability to: 
obtain receipts confirming referrals 
check quality and content of referred answers 
formalize their "cheat sheets" 
share references 

dynamically update reference guides 

obtain lists of their counterparts in other Centers 

ensure better access to FAQs and reference lists across Centers. 

Interestingly, most interviewees suggested that Specialists from various 
Centers meet in internal groups on a regular basis, because, currently, 
"Referents are in another office. They might as well be in a different world." 

These issues, and those above, will be linked to specific planning 
suggestion in Section 5 Recommendations. 
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5. RECOMMENDATIONS 



Recommendations are provided here for re-designing and improving 
ED.gov digital reference Centers. This section may be used as a planning 
document for re-designing the ED.gov digital reference Centers. 

Section 5.1 suggests how to choose a champion to implement the plan. 

Section 5.2 recommends selecting a specific level of centralization. 

Section 5.3 identifies important policy points to be decided by the 
centralized office. 

Section 5.4 presents a checklist for use in each Center to translate 
centralized policies into actionable processes and procedures. 

Section 5.5 lists software requirements that will inform the creation or 
selection of appropriate software. 

Section 5.6 advises the immediate and on-going analysis of existing 
software systems to inform a make/buy decision. 

Section 5.7 suggests an outline of training goals for managers and 
Specialists. 

Section 5.8 presents approaches for evaluating Center outcomes. 

Section 5.9 recommends that internal communication and customer 
feedback be organized to inform and update all of the above activities over time. 

Figure 3 Recommended Processes, illustrates these recommendations as 
steps in a chronological process: 




Figure 3 Recommended Processes 

Each arrow in the Figure represents a step and is labeled with the Section 
number that describes it in the sections below. 



; BEST COPY AVAILABLE 
o 

ERIC 
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5.1 SELECT CHAMPION OR SPONSOR 



ED. gov's digital reference Centers comprise a rich network of human 
expertise. That nehwork began as an informal effort when there was little traffic, 
but has grown along with the Web that supports it. Now the Centers are facing 
increased demand and must coordinate fast-growing information resources. 

Like all growing organizations the Centers need to re-examine their goals, 
determine which processes to formalize and how to expend assets. Each of 
these activities consumes resources and requires considerable coordination of 
disparate groups, thus requiring that a high-level executive serve as champion. 
The champion of such an effort would have authority over budgeting, personnel, 
and technology. 

The research team advises that the CIO of ED.gov champion the 
continuing development of digital reference Centers at the Department of 
Education. The team also advises that the CIO use the following 
recommendations to optimize the Centers' productivity and resource use. 



5.2 DETERMINE THE LEVEL OF CENTRALIZATION 



The champion's first task is to determine the optimal level of centralization 
from among four levels (illustrated in Figure 4 Levels of Centralization). 



Level 1 Level 2 Level 3 Level 4 
Highly Moderately Minimally Decentralized 

Centralized Centralized Centralized 




Figure 4 Levels of Centralization 







The optimal degree of centralization can be informed by considering four 
levels presented as scenarios below. Deciding upon the level of centralization will 
help determine how closely each Center must adhere to standardized 
procedures. 

Level 1: Highly Centralized 

One centralized Customer Service Department sets policy and 
standards, performs and monitors all ED.gov digital reference activities. 
Characteristics of highly centralized organizations are high levels of consistency 
in processes, an efficient flow of information from executive management to 
operations, impaired flow of information from operations on up, and inability to 
harvest expertise of individuals to improve the organization and its products or 
services. Because Level 1 centralization discourages the upward of flow of 
information, and Specialists' expertise is an important part of this plan, this level 
of centralization is not appropriate. 

Level 2: Moderately Centralized 

One centralized Customer Service Office sets policy and standards. 
Individual Centers perform services, monitor for compliance and report 
performance. 

This model makes good use of centralization to standardize shared tasks 
and general processes, yet allows Centers to capitalize on their most important 
assets - the expertise of their Specialists. Level 2, therefore, seems best-suited 
to the ED.gov digital reference Centers. 

Level 3: Minimally Centralized 

One centralized Customer Service Coordinator sets policy. Individual 
Centers interpret that policy to their own needs, set their own standards, and 
perform operations however they choose. One difference between Level 2 and 3 
is the locus of standards creation. Unlike Level 2, Level 3 centralization makes 
does not require monitoring for compliance or reporting outcomes to a 
centralized entity. This level imposes specific conditions (policies, standards and 
procedures) without the benefit of evaluating outcomes. Level 3 centralization, 
therefore is not an optimal choice. 

Level 4: Decentralized 

Level 4 describes the present organization of the Centers. Individual 
centers now decide whether to operate according to any policy, and can set their 
own policy and standards. They are not required to monitor for compliance or to 
report outcomes. While this model has allowed the evolution of rich and useful 
networks, networks operations are increasingly constrained by disconnects in 
software, priority, and resources sharing. 

In summary. Level 1 would prevent the knowledge of Specialists from 
contributing to the Centers' best practices and is therefore rejected. 

Level 3 requires that Centers adhere to policies set outside their domain, 
but does not evaluate if the Centers benefit from them. Level 3 is also rejected. 
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Level 4 is the current status of the digital reference Centers, and is not 
sufficiently supporting their tasks. Level 4 is rejected. 

The growing size, complexity, and importance of the digital reference 
Centers will be best served by Level 2 Moderate Centralization. The research 
team recommends that the champion or sponsor of digital reference at ED.gov 
adopt Level 2 and implement it using the following recommendations. All 
recommendations are made based upon the selection of Level 2 as the level of 
centralization and will support shared standards of quality across the 
Department. 



5.3 IDENTIFY POLICY POINTS 



The next step is to determine policy points that should be established by a 
centralized Customer Service Office (as per Level 2) and standardized from 
Center to Center. Policies should be established to dictate criteria for 
determining: 

(a) how to create templates for reacting to unanticipated current events 

(b) how to notify Centers in advance of public announcements 

(c) how to create communication devices for journalists to warn of traffic® 

(d) whether to increase frequency of Edinfo announcements regarding new 
developments(from a weekly to a daily basis) 

(e) if encryption or watermarks in answers could determine what publication 
has cut and pasted ED.gov information products 

(f) how to diminish, report and handle security threats 

(g) the identification of customer populations and their priorities of service 

(h) if Centers must use the same software applications 

(i) if Centers must use the same hardware platforms 

(j) a common protocol for sharing information across platforms 

(k) whether to use archives for historical preservation 

(l) devise procedures for answering various types of questions 

(m) suggest when to use various types of answers 

(n) specify required qualifications for Specialist positions 

(o) create career paths through advanced levels of expertise 

(p) the parameters of in-scope questions and strategies for managing them 

(q) minimum quality of information in answers, including response time (See 
Kasowitz and Bennett, 1999) 

(r) how to design, share and use FAQs and archives 
including how to: 

create naming conventions for FAQs 

keep FAQs accurate and up-to-date 

keep FAQ content consistent across related resources 

share FAQs across diverse platforms and media 



® The research team recommends the creation of a Current Events Response team. The Team 
should comprise Specialists, Program office public information functions and Web-designers. 
Together they can create tools and procedures to respond quickly to current events. 
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determine whether to use archives for historical preservation. 



The AskA Starter Kit: How to Build and Maintain Digital Reference 
Services (R. David Lankes and Abby S. Kasowitz) provides a helpful guide for 
considering these issues. Specific recommendations are made in Module 2, 
Planning. A copy is included with the original of this report. 

Policy points should be documented and, depending upon the degree of 
detail, included in updated versions of documentation at www.ed.gov/internal, 
including US Dept of ED-WWW Policy and Procedures , and 
www.ed.gov/internal/inet5.html . 

A useful tool for translating standardized policies into operation is the 
checklist, and its use is illustrated in the next section. 



5.4 CHECKLISTS 



This section presents two sample checklists to translate standardized 
functions into actionable processes and procedures. Each Center may decide 
upon slightly different ways to enact the policies listed above. The checklists 
below are therefore prototypes of devices that would allow each Center to ensure 
that centralized policy is guiding daily operations. Checklists 1 and 2 are meant 
as examples, only. Final checklists can be created after policy is set and 
feedback obtained. 
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Checklist 1 ANSWERING QUESTIONS 



ABOUT THE CUSTOMER 

Who is the customer and how should their answers be prioritized? 
Primary customer (e.g., the Secretary^ °) 

Secondary customer (e.g., the Program office) 

Other customers (e.g., the public) 

ABOUT THE QUESTION 

Is the question within scope? (definition of scope inserted here) 

If out-of-scope send it to (referrals here) 

If overload question send it to (referrals here) 

If security threat send it to (referral here) 

ABOUT THE ANSWER 
Does the answer: 

meet standard response time? (specify turnaround time here) 
meet the minimum standard of quality? (list criteria here) 
use the most appropriate format? (choose one of the following) 
citation 
pointer 
full text 
statistic 
referral 
research 

customized research 

synthesis 

compound. 

Should the answer be incorporated into a FAQ? (See FAQ checklist) 
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Italicized text represents possible answers for a specific Center and are examples, only. 



Checklist 2 CREATING AND MANAGING FAQS 



To contribute an answer to a FAQ, answer the following questions: 

Is this information already contained in or referred to in existing archives? 
Is this information already contained in an existing FAQ? 

Is the answer information indexed according to standards for easy 
retrieval? 

Is the answer contained in a file that uses standard naming conventions? 
Is the information accurate and up-to-date? 

Is the information in this FAQ consistent with other Centers' FAQs? 

Does the answer use language that is easily understood across Program 
offices? 

Is the FAQ accessible to various platforms and software programs? 



O 

ERIC 
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The two checklists above are examples of how to methodically transfer 
high-level policies of the Department into daily operations. Checklists should also 
be created to help translate all policy points into daily operations. 



5.5 SOFTWARE REQUIREMENTS 



Section 3.2.2 introduces the notion of scalability and the ability of software 
to support scalability. If the number of digital reference questions were to remain 
at current level, they could be answered by human intermediaries - Specialists. If, 
however, one accepts the assumptions that (a) digital reference Centers want to 
encourage customer-initiated communication, and (b) that most customers will 
have access to the Web, it becomes clear that some kind of automated 
intermediation will be required to facilitate communication. 



5.5.1 INTERMEDIATION 



Intermediation by humans is most effective at the level of "meta- 
intermediation", i.e., decision-making about the information to be provided, and 
the design of the software and systems that will provide it. 

Intermediation by software provides assistance at a lower level, 
conducting repetitive actions, transactions with individual customers. 

Thus, software-based and automated processes could leverage human 
expertise by freeing Specialists from repetitive tasks, to perform the uniquely 
human task of synthesizing knowledge in customer-oriented contexts. 

The domain of electronic commerce offers many kinds of software 
solutions that aid in scaling human expertise. They are described as software for 
digital reference Centers, e-mail management, message management, customer 
service, call Centers, response Centers, Customer Relationship Management 
(CRM) and "Ask-A^^" Centers. These software systems support customer e-mail, 
internal email routing, tracking, databases, and response management. 

Section 4.4 established that Centers must leverage human expertise 
through the use of software and mentions that such software is often 
referred to as Ask-A software. 

AskA software has been successfully used to coordinate information 
service organizations, but is recommended here for internal coordination, 
specifically for use as an intranet for ED.gov digital reference Centers. AskA 
software resolves centralization issues because it allows "front-line" Specialists 
to enrich archives and FAQs, and to contribute to the creation of standardized 
fields and operating systems. The remainder of this section describes: 
basic AskA functionality 
challenges to basic AskA software, and 
future functions for software within ED.gov. 



Ask-a Centers connect customers with questions to experts, as in "Ask-an-expert". 
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5.5.1. 1 BASIC AskA FUNCTIONS 



The following description is derived from AskA software functionality as 
reported in Section 1; Introduction, Virtual Reference Desk AskA Software, 
Decision Points and Scenarios, September 29, 1998, 
www.vrd.org/Tech/AskAsw-rpt.html. That section outlines functionality and 
specifications for AskA Software and provides a customizable workflow system 
for individual digital reference services. 

The system is based on a set of five Web-based modules that can be 
customized by each Center to best meet its individual requirements with 
maximum flexibility and ease of use. The five components of the meta- 
description question/answer process are defined in Table 3 Components of the 
Meta-Description Question/Answer Process. The following descriptions are 
based on assumptions made for Version 1 of the AskA software and do not 
reflect possibilities of all AskA services. 
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Component 


Description 


1. Question Acquisition 


A Web form allows a user to submit a question and other identifying 
information. During this process users may be prompted to identify a topic 
for their questions. Some automated pre-processing may occur to put the 
question in HTML or assign some sort of tracking identification. 


2. Triage 


A message is queued and prepared for expert response. In services with 
multiple experts, some automatic or human-mediated triage process occurs. 
(For example, Ask A Volcanologist uses a PERL script to simply send 
questions to on-call experts based on the days of the week, while MAD 
Scientist Network uses human moderators to select experts based upon 
their expertise. ) Human mediators may have permission to perform certain 
actions on questions before assigning them to experts (e.g., split messages 
containing multiple questions, delete inappropriate questions, etc.). 


3. Expert Answer 
Generation 


An expert generates an answer using personal knowledge and available 
resources. This component also includes any actions that experts may be 
allowed to perform on questions that have been assigned to them (e.g., 
make modifications to a question, send a question to a different expert, etc.) 
as well as "review" actions that other service staff can perform on completed 
responses as a form of quality control and/or expert training. 


4. Answer Sent 


Once an answer is generated, it is posted to a Web page. The answer may 
also be sent via e-mail if possible. Users may be able to follow up on 
responses directly from the answer page. 


5. Tracking 


Trends, subjects and other question/answer data from service transactions 
are tracked and used. In some cases the trends are used in Web 
development. Tracking can consist of Web-based archives, private archives, 
or simply informal information. 



Table 3: Components of the Meta-Description Question/Answer Process 

The meta-description constitutes the core of the AskA system. This core 
manages the information flow within an AskA service and provides fundamental 
system services (i.e., assigning tracking data to questions, managing a database 
of possible expert respondents, sending out responses to customers). It is, in 
essence, a database that supports the core workflow process and related 
functions in digital reference Centers. The core, however, is insufficient to 
capture all the complexity of a Center’s processes. The complexity of those 
challenges is described in the following section. 

5.5.1 .2 CHALLENGES TO BASIC AskA SOFTWARE 

This section presents a list of issues not addressed by the core AskA 
software requirements. These issues should be considered as requirements that 
will inform the creation or selection of software appropriate to ED.gov. The three 
most important challenges facing the selection of software are: 
convergence of information sources 
convergence of media types 
standardizing protocols across platforms. 
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5.5.1. 2.1 CONVERGENCE OF INFORMATION SOURCES 



Specialists at ED.gov are aware that many Centers are trying to resolve 
the same problems. In response they have formed the Frontline Forum - a group 
of Center administrators who meet to address issues. Most of the Department's 
Program offices participate in the Frontline Forum. They hope to promote 
communication, coordination, and sharing of information and techniques among 
ED'S telephone, email, and U.S. Postal mail call Centers. 

Similarly, USGS is trying eliminate duplication of effort, diminish customer 
confusion, reduce the number if repetitive dialogues, and prevent errors or 
discrepancies in answers across its many distributed customer service contact 
points. A sub-group (comprising EROD, VRD and IRC) is developing a referral 
database. 

It appears then, that the Centers are ready to develop and use processes 
and procedures to support broad interactivity and resource sharing. The danger 
here is that processes and procedures will spring up, as did the services they'll 
support, without including input from appropriate and interested Centers. 

Feedback from human beings - both customers and Specialists - is 
crucial to the success of ED.gov. But the feedback must be gathered in a 
methodical and inclusive way. 

The research team, therefore, recommends the formalization and 
sponsorship of the Frontline Forum. The Forum should receive funding to 
meet regularly with representatives from all Centers. Together they will 
resolve the issues raised in this report, and inform processes that resolve 
those issues. The Forum should report to the CIO on action items, new 
developments and ongoing efforts. 

Specifically, the Forum should contribute to 

(a) investigating software 

(b) implementing AskA and QuIP protocols, and 

(c) creating, monitoring and evaluating resource-sharing activities 
and products. 

These same issues should be addressed at annual Help Desk 
conferences, and presented at digital reference conferences and presentations 
for ED.gov staff and management. 

In addition to identifying best practices and determining useful metrics, 
participation in the Frontline Forum and Helpdesk conferences would enable 
Specialists to inform the creation of helpful activities and products, some of which 
are described here: 



5.5.1 .2.1 .1 DYNAMIC YELLOW AND WHITE PAGES 



A centralized database should be available to Center Specialists 
and administrators. It should list phone numbers and emails indexed by 
individual, by program, by project, and most important, by area of expertise. This 
database will serve across departments as an accurate and intuitive "who-knows- 
what" referral system, or internal locator, and should include personnel from all 
ED.gov Centers. 

This reference would require that personnel list their areas of expertise with 
Human Resources. The database can be updated dynamically as personnel files 
are updated with new job titles, and as people move from office to office. 

5.5.1 .2.1 .2 FAQs AND "SUGGESTION" SOFTWARE 

Finding answers to Frequently Asked Questions can be difficult, and many 
Specialists simply generate answers, create naming conventions and storage 
formats, and select storage locations for files, all from their own experience and 
according to their individual preferences. The standardization and coding of 
answers will help to organize them in an orderly method. But as individual 
Specialists attempt to cover growing domains of knowledge, they need intuitive 
ways to retrieve answer already created. "Suggestion" software is the name 
given to a set of applications that test a question for reserved words or phrases, 
and automatically offer topics that the Specialist can review, and from which he 
or she can select an answer. "Suggestion" software is featured in several 
commercial applications and in at least one not-for-profit AskA service, and is 
used to ensure consistency of answers across working groups. 

Many small databases exist across ED.gov Centers and contain 
answers to frequently asked questions (FAQs). All such databases should 
be culled for useful information which can be included in one large FAQ 
database for all Centers to share. Suggestion software should be run 
against all Center databases when offering answer solutions to 
Specialists. 

5.5.1 .2.1 .3 REFERRAL SOFTWARE 

Specialists are committed to delivering high-quality answers, and would 
like to ensure similar quality in questions they refer to other Specialists. 
To that end, referral systems should generate and send receipts of referred 
questions to the original Specialists. In addition, answers provided to customers 
by referred Specialists should also be sent to the original Specialist for quality 
inspection and inclusion in FAQs. 



5.5.1. 2.1. 4 



AUTOMATED INDEXES AND ARCHIVING FOR FAQS 



The Frontline Forum should catalogue all answer topics and assign to 
each a unique code. The appropriate code could be used as an internal "Dewy 
decimal" system, and should be included in every answer for easy and consistent 
FAQ development. This task must consider trade-offs between coding 
consistency and distributed effort, as well as optimal levels of granularity and 
depth. 



5.5.1. 2.2 CONVERGENCE OF INFORMATION SOURCES 

Within the Department of Education there are currently 190 active digital 
reference Centers. Multiple Specialists address questions for each Program, 
project, office and grant. The resulting repetition of answer-creation and 
diminished consistency of answers suggests that information sources be 
coordinated. Planners and managers should consult Specialists and: 

• decide upon multiple and redundant entry points into ED Info 

• reduce unnecessary or highly volatile entry points 

• offer one centralized access point for customers who need more 
prompting 

• return personal ownership of processes to Specialists. 

These points begin to address the convergence issues from both the 
customers' and the Specialists' perspectives, both of which are described below. 

5.5.1 .2.2.1 INFORMATION CONVERGENCE FROM THE CUSTOMER 
PERSPECTIVE 

The research team recommends that ED.gov allow customers to access any of 
its information resources through a centralized Web presence. This presence 
could be linked to, or represented by an icon on the Department's home Web 
page. The icon or linked page should allow customers to obtain ED.gov 
information from what looks like a seamless, one-stop access point. In actuality, 
the "back end" of the system will use QuIP and AskA software, and the unique 
coding scheme (mentioned in 5.5. 1.2. 1.4 and 5. 5. 1.2.2. 2) to distribute, track, 
gather, answer and archive answers for all the Centers. 

The key concept here is providing quick access to diverse resources 
through a simple interface. An example, albeit somewhat simplistic, is presented 
in Figure 5 Unified Information Access Point for ED.gov. 
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Welcome to www.ed.gov 
Click below for help with: 



HELP 



Topic oneofmany 
Topic twoofmany 
Safe & Driig-free Schools 



Evaluation and Feedback Button 



Save and Drugfree Schools 

Click be low for help with: 

____} Policy 

Preparing Children 

Evaluation and Feedback Button 



Preparing Children ^ 
Click below for help with: 



Children under 8 
Children over 8 
Help in the community 

Evaluation and Feedback Button 



HELP 



Figure 5 Unified Information Access Point for ED.gov 

Every effort should be made to accomplish the following goals for ease of 
customer navigation: 

1. Information sources should be available within 3 "mouse-clicks". That is, 
customers should not have to dig for deeply buried links to resources. 

2. Every screen should allow customers to access a general help tool, thus 
preventing customer frustration. 

3. Every screen should accommodate customer feedback, suggestions and 
complaints. 

5.5.1 .2.2.2 INFORMATION CONVERGENCE FROM THE 
ADMINISTRATORS’ PERSPECTIVE 

While customers will see a unified access point to all ED.gov information 
resources, each resource represents a separate Center. Centers are not located 
together, nor will they always use the same processes. Together, however, they 
must present a unified appearance to customers. The best way to accomplish 
that goal is to coordinate back-end processes that refer and track questions, and 
to share FAQs and archives. Back-end coordination will be aided by adoption of 
common policies and strategies. For example, archive and FAQ databases could 
provide tiered information sources. That is, databases could allow Specialists to 
enter data for local use, shared use with designated Centers, and global access. 



BEST COPY AVAILABLE 
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In summary, convergence of information sources should include 

• collection of feedback and coordination of standards through Frontline Forum 

• dynamically updated, shared reference guides 

• use of suggestion software with FAQ databases 

• automated reference tools 

• coded topics used for filing and retrieving FAQs 

• interfaces designed for ease of customer use, and 

• tiered administrative functions in databases and software. 

Coordination would be further aided by use of AskA software (see Section 
5.5.1), and QuIP standards (see Section 5. 5. 2. 3, below) and by addressing 
issues such as the convergence of media, and other technical issues. 

5.5.1. 3 CONVERGENCE OF MEDIA 

ED.gov digital reference Centers receive questions and provide 
answers via several communication media, including U.S. postal mail, 
telephone, fax, email and Web forms. Questions are generally tracked by 
the kind of media used, and their answers are rarely shared across 
communication modes. Because they cannot currently be tracked, these 
answers are continually re-created and waste resources. 

Some software features the ability to track multiple modes of 
communication and should be incorporated as needed across ED.gov 
Centers. While the number of phone requests is declining, for instance, it 
must be assumed that there will always be some phone-based dialogue. 

Further investigation will determine what percentage of resources should 
be devoted to non-email-based questions and answers in ED.gov Centers. 

Software problems have been identified by customers and by the 
research team. Potential solutions lie in AskA software packages, but 
which packages, and what features? The research team recommends that 
ED.gov research commercially available packages to determine if one or 
more could support its needs. This activity is described in the next section. 

5.5.1 .3.1 STANDARDIZING PROTOCOLS ACROSS PLATFORMS: QuIP 

Section 3.4.2. 1 states that fragmented approaches in tool design and 
tracking systems make it difficult to share tracking information and solutions 
across Centers. Section 4.6 reported that Specialists need shared tracking and 
answering systems. Such shared functions require a standard protocol for 
exchanging information across platforms. 

The research group recommends the application of QuIP (Question 
Interchange Profile). The QuIP protocol was developed at the Virtual Reference 
Desk (Information Institute of Syracuse^^). The following explanatory material 



’^The Virtual Reference Desk is a project headed by the ERIC Clearinghouse on 
Information & Technology. It is funded by the U.S. Department of Education’s 



was excerpted from Question Interchange Profile . White paper, Version 1.01D, 
1999. R. David Lankes, Director, ERIC Clearinghouse on Information & 
Technology. The White Paper demonstrates the viability of implementing a large- 
scale information system that directly utilizes human expertise. The White Paper 
is available in its entirety at http://www.vrd. 0 rg/Tech/QulP/l.OI/l.Old.htm. 

5.5.1. 3.1.1 QuIP BACKGROUND 

The Question Interchange Profile (QuIP) was a concept first proposed in 
the Virtual Reference Desk’s White Paper (Lankes, 1998a) and research report 
on K-12 digital reference services (Lankes, 1998b). It was subsequently 
discussed in Virtual Reference Desk meetings and AskA consortium meetings. 

QuIP was proposed as a means to distribute questions and answers 
among K-12 digital reference services (AskA services), and has subsequently 
been extended as a generalized means of expressing discussion-like 
interchanges between one or more people or organizations. 

At the heart of QuIP development is the belief that reference interchanges 
can be both computational (that is, able to be processed by software without 
human intervention) and extensible (into various domains for example) while 
retaining the value of human to human communication. 

This following description presents QuIP in the abstract. It is envisioned 
that QuIP will be transmitted via HTTP, SMTP, NNTP, Z39.50 and other 
protocols appropriate to specific contexts. QuIP is a profile that sets definitions 
beyond a simple set of elements. Some elements are mandatory and repeatable, 
others are pre-defined values and may not work in all circumstances. Variations 
of these choices using the same base of elements represent different profiles. 
Development was conducted in such a way as to minimize the need for multiple 
profiles, but it is not ruled out at this point. 

5.5.1. 3.1. 2 QuIP FOUNDATIONS 

The heart of QuIP is a thread - an exchange of question(s) and answer(s). 
Exchanges may span many interactions, with changing subjects and personnel. 
Thus, a thread can be a shifting, on-going dialog. A simple thread can simply be 
a question, followed by an answer: 

Customer: How many senators are there in the congress? 

Specialist: 100, two from each state. 

On the other hand, in the face of ambiguity, a thread may consist of several 
interactions: 

Customer: How many senators are there? 
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Specialist: In state government or in the federal government? 

Customer; The federal government. 

Specialist; 100, hNo from each state. 

As illustrated in the second example, questions can come from both 
customers and Specialists. The difference, however, is that a customer is asking 
a content related question, while the Specialist is asking a clarifying question. 
These clarifying questions can be open-ended, or closed in nature. 

Perhaps the best definition of a thread is as a series of interchanges. 
These interchanges have several attributes. For example the interchange has 
content - the question, or answer, or some other informative data. It also has a 
date such as when it was sent, received, or acted upon. Each of these attributes 
also can have sub-elements, or attributes themselves. For example, the content 
of an interchange has the text of an answer, or question, but it also has a format 
that that text is expressed in, and a language that expresses it. For example, the 
content of the interchange may be an HTML file written in English such as: 

<html> 

<head><title>The Answer</title></head> 

<h1>The answer is....</h1> 

<p>100, two from each side.</p> 

</body> 

</html> 

Since most of the processing and use of QuIP will be automated, this level 
of detail and metadata is necessary. (QuIP must be as specific and succinct as 
possible to allow for computational manipulation.) 

Other elements, attributes, sub-elements and sub-attributes are fully explored 
in the Question Interchange Profile White Paper. The purpose of this section is 
only to present the fundamental concepts of the profile. 

QuIP is hierarchical in that, for example, a thread may have one or more 
interchanges, and interchanges can have one or more dates. Another important 
concept is that some elements of QuIP are mandatory, repeatable, and some 
attributes of these elements are pre-defined and extensible. These concepts are 
explained below; 

• Elements in QuIP are either mandatory, conditional or optional: QuIP must 
have (in fact is) a Thread. A Thread must have one or more interchanges. 
An interchange may or may not have restrictions. While a Person element 
is optional. If used, however, the person element must have an associated 
Surname. 

• Elements in QuIP are either repeatable or not; You can only have one 
thread, however a thread can have multiple interchanges (in fact the 
simplest example of a thread would have two interchanges: an originating 
question and an answer). 

• Elements in QuIP can have pre-defined values. In some cases there is no 
limit to the number of values an element can contain. In some cases. 
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however, there is an established range of values an element can take. 
These ranges can be relatively small and controlled like the state of an 
interchange (it can have one of five values: new, assigned, answered, 
sent, closed). Ranges can also be expansive, but controlled like the 
language of an interchange. 

• Elements and attributes can be extensible: In order to make QuIP 

accommodate existing practices and software, key elements of QuIP are 
extensible: QuIP has a place holder where other metadata standards and 
definitions can be used. For example, it is anticipated that digital reference 
services will use a variety of subject schemes and thesauri. Rather than 
selecting one, QuIP allows a service (or a Center) to specify a vocabulary 
to use, and the language of that vocabulary, and a place to look up that 
vocabulary. This means that a service (or Center) can use any subjects, 
so long as they indicate which ones they are using, and how a computer 
can make sense of them. 

5.5.1 .3.1 .3 QuIP IS PROCESS ORIENTED 

A foundation of QuIP is that it is process oriented. QuIP is designed to 
transmit data from one Center to another. It is the exchange that drives QuIP, not 
the actual content of the exchange. This means that Qu IP-formatted threads fully 
formed document like objects. While QuIP could be used as an internal file 
structure of digital reference services, it is assumed that most of the time 
services will parse QuIP data and transform it into another format internal to a 
service (such as a relational database). This assumption influences the predictive 
nature of QuIP and is the reason for QuIP's structured definitions. If the format of 
the data is known in advance, then programs can be written to manipulate the 
data without human intervention. (This concept is fundamental to metadata, but 
space precludes a general discussion on that topic, here.) 

5.5.1 .3.1 .4 QuIP INSTIGATORS 

A key concept in QuIP is that interchanges are initiated by a process or 
person and that entity is called an instigator. Threads have interchanges that 
come from an instigator. Instigators can be a people or an organizations. The 
instigator has a profile that describes the entity originating an interchange, and a 
role that defines what that entity is doing. Note that an interchange assumes 
there is both the instigator of a message and a receiver of that message. In the 
QuIP element set, the receiver is implicit, and no elements define the receiver. 

An interchange is one way, with interactivity (a discussion) coming from iteration 
to form a thread. A more complex thread is made up of multiple instigators and 
multiple interchanges. For example, the receiver in one case, can become the 
instigator in another. 

5.5.1. 3.1. 5 QuIP RESTRICTIONS 
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Several other fundamental concepts will support a full understanding of 
QuIP. The first is that interchanges have a sequence, a subject and a set of 
restrictions. Restrictions are mechanisms for building complex types of 
transactions with interchanges. For example, one type of restriction is a 
certification. 

With this mechanism services can build tools to force Specialists to have 
some level of knowledge (such as an MLS in a library world, or CNE in a help 
desk environment). These certifications can then be matched to qualifications 
(part of the instigator information). Other Restrictions can also be expressed to 
enforce Center policies and standards ( e.g., this question must be answered in 
two days). 

5.5.1.3.1.6 QuIP IDENTIFICATION AND REPOSITORIES 

Two final concepts are key. One is the concept of identification. The other 
is the implicit dependency on repositories. In order to manipulate QuIP data 
computationally, unique identification of data is mandatory. 

Identification is based on QuIP Unique Identifiers (QUID). A QUID is 
composed of two elements. The first is a QulP_Registy_ID. This ID is unique to 
a given digital reference service. Thus, a service like AskERIC might be given a 
registry ID of ‘ASKE’. The LocalJD is a number that is assigned by, and unique 
within a digital reference service like 0000001. A QUID, in this case might be 
ASKE-0000001. This number will follow a thread no matter how many services 
may handle or hand-off a thread. With a QUID, one can uniquely identify an 
interchange within a thread by a combination of a QUID and the sequence 
information associated with a given interchange. 

There are several pieces of information that cannot be contained within 
the QuIP specification, but are required by QuIP data. For example, 
QulP_Registry_ID’s must be unique, and the number of entries will grow as new 
digital reference services come into existence. Profiles and vocabularies are 
other examples, and are discussed further in the White Paper. 

5.5.1. 3.1 .7 QuIP ELEMENTS 

Table 4 Simplified QuIP Elements, is a simplified version of first table in 
the White Paper, and contains elements and definitions, only. 



Element Name 


Definition | 


Affiliation 


A non-person organization (company, professional association, etc.) to which a j 
person belongs as an employee, volunteer or member. EXAMPLE: Library of | 
Congress | 


Certification 


An abstract representation of a skill or level of mastery. These may include I 

degree designations or professional demonstrations. EXAMPLES: MLS, CNE | 


Content 


The data being interchanged | 


Date 


A soecific ooint in time exoressed in Months. Davs and Years. EXAMPLE: ] 
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01012000 


Dates 


Information in an interchange relating to when an interaction occurred, or was 
in some way manipulated (such as a change in a state). 


DateType 


A label identifying a specific type of manipulation of an interchange. So that a 
service can record when a message was received, answered or sent. 
EXAMPLE: Sent 


Description 


An abstract of the content of an interchange. Such a short description can be 
used in retrieval or browsing. Analogous to the subject line of an e-mail 
message. EXAMPLE: How Many Senators are there? 


E-Mail 


An electronic mail address of a person or organization. EXAMPLE: 
bob0).example.com 


FirstName 


A person’s first name. EXAMPLE: Richard 


Format 


The file type of a document or text. This information is necessary to de-code 
and manipulate information being exchanged. EXAMPLES: HTML, ASCII, GIF, 
MP3 


ID_String 


An identifier that can be used as a label or an input to a profile repository. 
EXAMPLE: rdlankes 


Instigator 


The person or organization that originated a given interchange. So when a 
user asks a question, he or she is the instigator. However, when a service or 
Specialist responds to that original inquiry, the Specialist or service is the 
instigator. 


Interchange 


The basic repeating element of a thread. It contains the information that varies 
as a thread changes hands (from service to service, or user to service). 


Language 


The language used to encode information. EXAMPLE: English 


LevelSpecialty 


Designation from a given vocabulary of non-topical oriented specialty that 
indicates position in a hierarchical, or continuous system. This element is used 
to indicate either a nominal level such as grade in a school context, or ordinal 
such as trainee. EXAMPLES: 6, Primary, Manager 


LocalJD 


An ASCII string that represents a unique identifier within a given service. 
Combined with the QuIP Registry ID it uniquely identifies a thread regardless 
of the service. EXAMPLE: 00000000001 


Locator 


The means of identifying the location of a person or organization on the 
Internet or in real space. 


MiddleName 


A person’s middle name or initial. EXAMPLE: David 


Organization 


An aggregation of individuals or an abstraction that represents a non-person 
such as a company, corporation, or professional organization. EXAMPLE: 
National Library of Education 


Org. Name 


Name of an organization. EXAMPLE: National Library of Medicine. 


OtherRestriction 


Information used to control the use of an interchange. Restriction data can 
relate to service level agreements on time to answer, commerce data, or any 
other data that impacts how and why a question changes service ownership. 
EXAMPLE: 2 day turn around. 


Person 


A human being that initiates an interchange. The person can be a user with an 
information need that initiates a thread; a Specialists who has the knowledge 
to meet an information need, or an intermediary that brokers an exchange 
between a user and a Specialist (see role). 


Postal 


A postal address. EXAMPLE: 4-192 Center for Science & Technology, 
Syracuse, New York, 1 3244 


Prefix 


Title or designation used in a name. EXAMPLE: Dr. 


Profile 


An abstract container for information relating to a person or organization that 
initiates an interchange. 



ProfileJD 


In order to be extensible the ProfileJD represents a mechanism to refer to 
external profile information. These external profiles can be domain specific or 
simply enriched beyond the QuIP person and organization scheme. 
ProfileJDs can also be used to create a local ID that are only referenced 
within a thread, and minimize repetition of profile information through multiple 
interchanges. 


Profile_Location 


A means of locating profile information. Locations may include a URL to an 
online profile repository, an XML file, or some other means of computationally 
extracting person or organization data. 


Qualifications 


An indication of a person or organization’s ability to perform in a given role. For 
example, a Specialist’s ability to answer a question. 


QUID 


A unique identifier for a thread. 


QulP_Registry_ID 


A string that represents a digital reference service uniquely. 


QuIPThread 


The primary element of QuIP, it is defined as a series of interchanges with a 
single identifier (QUID). This is the root element of QuIP. 


Restriction 


A means to control the use of an interchange. This element is intended to be 
extensible to a given domain such as e-commerce. 


Role 


An abstract representation of an instigator’s position or stake in an 
interchange. This role determines the information about an instigator needed 
beyond basic identification. 


Scheme 


A information organization system that can be used computationally to 
disambiguate information, or map one system to another. EXAMPLES: GEM, 
IMS, MARC 


Sequence 


An ascending real number that represents the order of interchanges. 
EXAMPLE: 1. 


State 


An indication of the current action being conducted on an interchange or the 
present condition of an interchange. 


Subject 


The topic of a question. EXAMPLE: Science. 


SubjectSpecialty 


A topic in which a Specialist has extensive knowledge and understanding. 
EXAMPLE: Science. 


Surname 


A person’s last name. 


Telephone 


A phone number used to contact a person or organization. 


Text 


The body of a message. Text here is used in a broad sense as in analysis of 
the text of a book. This is the original creation of an instigator that is in the 
format of the content of an interchange. EXAMPLE: How many senators are 
there? 


Type 


The role being played by the instigator of the message. The role can be as a 
USER, that is a person or organization with an information need; a 
SPECIALIST, that is a person or organization with extensive knowledge and/or 
experience to alleviate on information need; or an INTERMEDIARY, that is a 
broker that matches a user with a Specialist. 


URL 


A Uniform Resource Locator used to identify the Internet location or an 
electronic resource. EXAMPLE: http://www.vrd.org/ 


Vocabulary 


A cohesive collection of terms that identify key concepts in a given domain. 
EXAMPLES: the ERIC Thesaurus, GEM Subjects. 


VocabularyEntry 


A term in a vocabulary. 


VocabularyName 


A label used to refer to a vocabulary. 




Table 4 Simplified QuIP Elements 
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EXAMPLE 



The following is a simple example of data represented by QuIP for a single 
interaction: 

QUID; ASKE-000001 

SEQUENCE: 1 

STATE; NEW 

DATE_TYPE: ORIGIN 

DATE: 01012000 

FORMAT: ASCII 

LANGUAGE: English 

TEXT: How Many Senators are there? 

SUBJECT; Senators 
TYPE: User 
PROFILEJD: Dave 

Minimal QuIP Information (simple exchange) 

QUID: ASKE-000001 
SEQUENCE: 1 

STATE: NEW 

DATE_TYPE: ORIGIN 

DATE: 01012000 

FORMAT: ASCII 

LANGUAGE: English 

TEXT: How Many Senators are there? 

SUBJECT: Senators 
TYPE: USER 
PROFILEJD: Dave 
SEQUENCE: 2 

STATE: ANSWERED 
DATE_TYPE: SENT 
DATE: 01022000 
FORMAT: ASCII 
LANGUAGE: English 
TEXT: 100, 2 from each state. 

SUBJECT: Senators 
TYPE: SPECIALIST 
PROFILEJD: AskERIC 

In summary, recommendations for software requirements address both 
human processes and software functionality, and include that the Centers: 

• use meta-triage approach and AskA software 

• create a convergence of information sources and media types 

• provide tools such as dynamic references and automated FAQs indexes 

• use suggestion software to recommend FAQs to Specialists, and 

• incorporate QuIP to standardize protocols across platforms 



5.6 MAKE OR BUY DECISION 
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Based on their software requirements, the Centers must choose between 
existing software and creating custom software. This section advises the 
immediate and on-going analysis of new software systems to inform a make/buy 
decision. ED.gov can choose from two categories of software. 

One category of software is small applications that run "on top" of existing 
Center software. That is the model currently in use and it is useful to the degree 
that it accommodates legacy systems. A major shortcoming is that such systems 
limit information sharing. Research into such software is not advised. 

A second category to consider is new systems. They also present 
challenges, such as large investments of time and money, and steep learning 
and implementation curves. Cognizant of these issues, the research team 
advises investigating new systems because they allow the sharing of expertise 
across Programs, office and departments that is crucial to ED.gov. 

Appendix E; Software Digest is a snapshot of packages that are available 
at this writing. The list is lengthy, and does not include details e.g., licensing 
options, prices, and which packages are customizable) - an indication of the 
large scope of commercial software available. 

Should existing software packages prove inadequate, software should be 
created to support the activities listed above, and to support future needs of the 
Centers, as mentioned in the following section. 



5.6.1 SUGGESTED FUTURE FEATURES FOR ED.GOV SOFTWARE 



Software will help Centers answer queries and collect extensive data 
about customers(customer relationship management software) 

In addition, future uses of software should include database mining, 
analyzing customer service feedback, and determining best new uses for the 
Centers' services. Difficult choices will have to be made, however, in deciding 
which functionalities to automate. 

As the use of ED.gov Centers grows, they will face the issues of 
language translation. It would be advisable, therefore, for ED.gov planners 
to begin (a) considering future policy about interpreting languages, and (b) 
determining if software is available for that feature. 



5.6.2 SUMMARY 



The specification, design, selection, implementation and 
evaluation of software are critical to the scalability and success of ED.gov 
Centers. Resources need to be allocated to these tasks, and careful 
consideration given to their outcomes. 

In addition to automating existing procedures, software can be used 
to perform database mining, to analyze customer service feedback, and 
determine best new uses for the Centers' services. These features should 
be incorporated into future uses of the Centers' software systems. 



5.7 TRAINING 
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The redesign of ED.gov digital reference centers and their 
processes will result in: 

new decisions affecting managers and administrators 

new software, processes and procedures that affect Specialists 

new interfaces that affect customers. 

The remaining three parts of this section suggest training for all 
three groups. The AskA Starter Kit: How to Build and Maintain Digital 
Reference Services (R. David Lankes and Abby S. Kasowitz) provides a 
thorough guide for planning training activities (see Module 3 Training). The 
Information Institute of Syracuse can make suggestions for more specific 
training, including training classes, on-site sessions and other formats. 



5.7.1 TRAINING MANAGERS 



Center managers need training that will prepare them for the following 

tasks: 

(a) learn how to translate policy points into Center procedures 

(b) be familiar with referral, out-of-scope questions, answering, FAQ and 
archiving issues 

(c) assign sharing levels to FAQs (personal, shared, or global) 

(d) determine whether to use archives for historical preservation 

(e) monitor the status of referred questions 

(f) notify Centers in advance of public announcements 

(g) react to unanticipated current events 

(h) use procedures to diminish, report and handle security threats. 

(i) create checklists to determine procedures for the following issues 

(j) coordinate communication between journalists and Specialists 

(k) participate in Frontline Forum activities 

(l) monitor coding of answers that indexes them for FAQs 

(m) be able to instruct Specialists in use of QuIP 

(n) create new staffing models to select and train staff to rotate responsibilities 

(o) plan backups (cultivate Specialist's domain expertise to share) 

(p) budget for resources, especially telephony support and training specialists 

(q) deal with international support issues 

(r) direct records management 

(s) ensure consistency across Centers 

(t) incorporate redundancy (preserve government digital documents that may 
otherwise disappear). 



5.7.2 TRAINING SPECIALISTS 



Specialists will benefit greatly from the re-designed policies. 







centralized resources and automated processes mentioned in previous 
sections. Each of these, however, will require that Specialists acquire new 
skills, including how to identify and deflect out-of-scope questions, such as: 

(a) technical email (intended for Webmasters) 

(b) requests to update sites/upload files (from Program offices and 
potential link partners) 

(c) error messages (autoresponses such as "server down") 

(d) maintenance (email from customer reporting server difficulties) 

(e) listserv email. 

Specialists must also learn to differentiate and choose from various 
answer types, including; 

(a) citations (name of a resource material) 

(b) pointers (name of resource material and instructions for accessing it) 

(c) full text (text from a resource material) 

(d) statistics (data with minimal context, usually numeric, brief answers) 

(e) referrals (notice that the question has been sent to another Specialist) 

(f) research (list of citations from ready reference materials) 

(g) customized research (citations/pointers resulting from detailed search) 

(h) synthesis (any of the above with explanatory verbiage) 

(i) compound (any combination of the above responses). 

Specialists must cultivate managerial skills so they may: 

(a) understand policy, standards and procedures, including tracking, archiving 
and referring questions 

(b) educate customers of varying expertise 

(c) decide when to teach customers how to use tools. 

(d) learn new software and tools including databases and automated processes 

(e) monitor and update knowledge management systems 

(f) prepare for flexible, shifting responsibilities 

(g) learn and maintain good writing skills 

(h) understand and employ proper "netiquette" 

(i) respond to security threats 

(j) aspire to qualifications for higher-level positions. 

Training should include incentives (to be determined after skill sets are 
identified), provide skills for advanced career paths and allow Specialists to 
contribute their expertise to the upgrading of ED.gov systems. Further work 
is needed in the planning and development of training, and should be 
carried out in concert with the creation of policy, standards, and software. 



5.7.3 TRAINING CUSTOMERS 



Customers have already learned how to access and navigate Websites, use 
email, and phrase queries. ED.gov should provide online training for customers 
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who want to want to perform more sophisticated searches. Such training would 
include information about how to: 

(a) use strategies specific to ED.gov for more sophisticated searches 

(b) respond to Specialists' messages (customers are instructed to send 
messages to another party, but often re-send to original Specialist). 

(c) evaluate legitimacy of online sources (if not ED.gov) 

(d) suggest improvements to the Centers. 

In summary, redesign of the Centers' goals, processes, and software will 
require that both managers and Specialists be trained. New functions will add 
unfamiliar tasks and material to Specialists' responsibilities. Customized training, 
strategically administered will require additional resources, but will result in 

(a) shorter learning curves 

(b) reduced error and 

(c) increased productivity. 

In short, the resources allocated to training will directly increase the Centers' 
scalability. 



5.8 EVALUATION 



Ongoing monitoring can evaluate adherence to policies, standards for 
customer satisfaction, improvement in internal processes and human resource 
issues. The AskA Starter Kit: How to Build and Maintain Digital Reference 
Services contains specific suggestions for evaluating activities. 

Checklists, similar to those presented in Section 5.4, should be used to 
monitor and evaluate the following topics: 



Adherence to policies and checklists: 

Have we provided appropriate answers? 

Have we identified our primary customer? 

Have we answered the right questions? 

Have we determined and adhered to the parameters for determining out- 
of-scope questions? 

Customer service: 

Have we provided faster response time? 

Have we provided consistent information? 

Have we provided accurate information? 

Have we provided current information? 

Have we used the appropriate answer format(s)? 

Internal processes: 
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Have we reduced question overload? 

Are file naming conventions consistent? 

Is tracking coordinated across media? 

Have we reduced technical email (intended for Webmaster)? 

Have we reduced requests to update site/upload files? 

Have we reduced the number of error messages? 

Have we reduced maintenance email (re: server difficulties)? 

Have we reduced out-of-scope questions of all kinds? 

Are FAQ files easily be retrieved and re-used? 

Are FAQ files are re-used and shared across Centers? 

Is FAQ information is complete? 

Is FAQ information is accurate? 

Is FAQ information is consistent with other official resources? 

Do FAQ naming conventions conform across Centers? 

Are plans in place to manage traffic-spikes that result from current events? 
Can cut-and-paste journalism be easily tracked to sources? 

Have we coordinated press releases and media events with Specialists? 

Human resources: 

Have incentives and career paths been created for advanced levels? 

Have qualifications been specified for Specialist positions? 

Have we identified and implemented ways to accomplish 
scalability? 

ED.gov Centers must reach satisfactory performance standards by 2001. 
That means that at least 90% of customers, internal and external, must agree 
that ED products, services, and information, including those on the Department's 
Website, are timely, accessible and of high quality. The evaluation topics listed 
above provide a guide to measuring Center outcomes. 



5.9 FEEDBACK 



Feedback is the richest source of focused input about practical 
improvements to a system. Every major topic mentioned in the Recommendation 
section depends, to some degree, on feedback from Specialists and customers. 
(The recommended Web interface (Section 5. 5. 2. 1.1), for example, advises that 
each screen solicit customer feedback.) 

Feedback is represented in Figure 3 Recommended Processes, as the 
last step in re-designing ED.gov Centers, but feedback occurs all throughout the 
entire process, and should be continuously gathered. 

Conversations, office meetings and email are all sources of feedback. 
More formal feedback can be gathered at Frontline Forum meetings, and through 
polls and surveys. 

Similarly, it would be advantageous to formalize some monitoring. For 
example, individuals could be asked to monitor and report back to the Centers 
about such matters as; 
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new legislation, 

the availability of multi-language software 
the progress and potential use of Internet II. 

Use of formalized feedback can be designed and implemented, 
however, only after policy points have identified specific and useful topics. 




79 



6 SUMMARY 



This report is a reflection of findings from the research into ED.gov digital 
reference Centers (Sections 1 through 4). It is also a planning document that 
allows for accountability and supervision, and enables Specialists to enrich the 
Centers with their expertise (Section 5). 

The planning document provides suggestions to provide consistent and 
fast operation across Centers and provides general recommendations for 
determining policy and software issues including that the ED.gov digital reference 
Centers and their managers; 

• choose a champion in the CIO's office to procure resources and 
determine the level of centralization (Section 5.1) 

• select Level 2 centralization to coordinate policy and standards (5.2) 

• incorporate AskA software into a Department-wide Intranet (5.5) 

• employ QuIP protocols to enable resource sharing and tracking, 
archiving and FAQs across Centers (5.5) 

• use checklists to translate policy into actionable items (5.4) 

• formalize the Frontline Forum and use it to coordinate software 
specification and standardize operations across Centers (5. 5. 1.2.1) 

• coordinate a "Fast-response" team to provide fast and accurate 
answers to questions about current events, thus preventing traffic 
spikes in the Centers 

• research commercially available software packages to determine if 
they support Center processes and procedures (5.6) 

• create training goals and plans, and decide on implementation 
mode(s) (5.7) 

• evaluate daily operations using checklist (5.8) 

• continuously gather and use feedback to upgrade systems and 
services (5.9). 



These recommendations are designed to better interrelate and redesign 
the ED.gov digital reference Centers. 

This research project has, in accordance with Educational Excellence for 
All Children Act (1999) created: 

models or applications that use technology to... promote the sharing of 
examples of promising practices developed under this authority in order to 
bring effective models to scale (page 76) (and) leveraged resources and 
promoted high quality (page 65). 
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As prescribed in the U.S. Department of Education Strategic Plan, 1998- 
2002 . the report provides guidance regarding: 

• training for front-line employees and planning workforce development 

• supporting strategic partners 

» providing seamless service based on optimal technological 
approaches 

• capturing and incorporating customer feedback, and 

• establishing standards for internal and external operations. 



Appendix A Glossary of Terms 

Archiving. The storage of the questions’ answers either in databases, online sites, text files 
or in hard copy. Archived information may be used to create better online information 
resources, including Frequently Asked Questions files (see FAQs). 

AskA Services. Internet-based question and answer services that connect users with individuals 
who possess specialized subject or skill expertise. Also known as digital reference services, they 
provide human expertise through question/answer services on the Internet. So called for services 
such as Ask-A-Scientist, these services take questions through e-mail and the World Wide Web. 
(Version 1 of the AskA software considers questions submitted through the Web only.) 

Center. Digital reference services that are under the aegis or the U.S. Department of Education, 
or of related organizations (see Digital Reference Services). 

Customer. Individual from any of the Centers’ many populations who submits a question to a 
Center. Customer populations include, but are not limited to: students, parents, researchers, 
journalists, attorneys, and school administrators. 

Digital Reference Services. Internet-based question/answer and referral services 
that reach a wide audience on a wide range of topics. Can include AskA services, library 
electronic reference services, help desks, and others. 

FAQs are teaching tools presented in the format of questions and answers. Their content 
is derived from archived answers, and is accessible to customers through a Web site 
interface. 

Offload. An expert’s capacity for receiving questions (e.g., 3 questions per week, 1 question per 
day, etc.) is sometimes exceeded. The excess, unanswered questions must be assigned to other, 
available specialists. That process is known as offloading. 

Question. A single information problem that can be solved by a Specialist. 

Specialists. Individuals who answer user questions submitted to U.S. Department of Education 
digital reference Centers. They possess special skills and knowledge derived from training, 
education, and experience. Responsibilities and functions vary from Center to Center (see 
Center). 

Thread. A set of related questions and answers that are stored in a knowledge 
base. Questions and answers can be linked in a thread by ID#, subject or other 
data. 



Tracking. The monitoring of a question's progress and status until the answer is sent to the 
customer. 



Appendix B - Notification of Poll 

IMPORTANT DEPARTMENT SURVEY - PLEASE RESPOND 

The Department of Education is conducting a study of e-mail customer 
service and coordination. As part of this study, several thousand e-mail links and 
publicly available e-mail addresses have been identified on the Department's 
web site. This short survey is intended to catalog these services and guarantee 
the continued availability of their e-mail addresses. 

Please respond to the survey by completing the web form at the URL 
listed below. You will be asked to: 

- identify individual(s) who monitor the email address (name, office, 

individual email, phone) 

- describe average weekly volume of email received 

- tell approximately what percentages of questions are 

- answered discarded referred to others (and list the top three referents) 

- specify the intended scope/nature/topic of questions designed to 

- be answered by your service 

- describe the actual nature/topic of questions received - characterize 

what trends (if any) you see forming in your service. 

Please respond to the survey within 10 working days (by September 21 , 
1999). Non-responses will be assumed "dead-links." That is, if you don't respond 
to the survey, we will assume that the address is not being monitored for 
customer mail and we will initiate removal of links to that address from the Ed. 
web site. To respond to this survey, please access: 
http://aska.syr.edu. 

Thank you, 

Keith Stubbs 

Director, Resource Sharing and Cooperation 
National Library of Education 



Appendix C: Polling Instrument 

DEPARTMENT EMAIL SURVEY FOR ED.GOV 

If you have received a message entitled "Important Department Survey", please respond to this 
survey within 10 working days (by September 21, 1999). All non-responses will be assumed "dead-links.” 
That is, if you don’t respond to the survey, we will assume that the address is not being monitored for 
customer 

mail, and we will initiate removal of links to that address from the ED web 
site. 
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This survey is being conducted by The Department of Education to study e-mail customer service 
and coordination. Several thousand e-mail links and publicly available e-mail addresses have been identified 
on the Department’s web site. 

This short survey is intended to catalog these services and guarantee the 
continued availability of their e-mail addresses. Please contact vrdsurvey@vrd.org if you have any problems 
using this form. 

If you have multiple active email addresses, please fill out the survey form 
once for each. 

Complete the following fields and submit your response. 

Thank you 
Keith Stubbs 

Director, Resource Sharing and Cooperation 
National Library of Education 

Last Name * 

First Name* 

Title 

Address 

City 

State 

Zip 

Email Address* 

Phone Number- - 

Email Software 
Don't Know 
Pine 
Elm 

Microsoft Outlook 
Microsoft Internet Mail 
Microsoft Exchange 
Netscape Mail 
Eudora 
Other 

Total Number of E-mails Received Weekly* 

Percentage Answered®/© 

Percentage Discarded®/© 

Percentage Referred®^ 

What department or individual is at the top of your list for 
referring questions? (example: Jane Doe at the National Library of 
Education, jane_doe@ed.gov) 

What department or individual Is second on your referral list? 

What department or individual is third on your referral list? 

What is the INTENDED scope of this service? 

How does the ACTUAL scope of email received, differ from the intended scope o 
of the service? 

What trends (If any) do you see forming in your service? 

Thank you for your time. 

Required fields are noted with an * 



Appendix D Database Record Structure for Poll 



Field 


Field 






# 


Name 


Description 


Characteristics 


1. 


LAST NAME 




20 characters alpha 


2. 


FIRST NAME 




10 characters alpha 
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3. 


TITLE 




40 characters alpha 


4. 


OFFICE ADDRESS 




80 alpha/num 


5. 


EMAIL ADDRESS 




50 alpha/num 


6. 


PHONE NUMBER 




10 char, numeric 


7. 


WEEKLY 


Estimated number of 
weekly e-mail questions 
received 


6 characters numeric 


8. 


ANSWERED 


Estimated percent of 
question the respondent 
answers 


2 characters numeric 


9. 


REFERRED 


Estimated percent of 
questions referred out 


2 characters numeric 


10. 


NAME1 


Name of person/office to 
whom question are referred 


60 characters alpha 


11. 


NAME2 


Name of person/office to 
whom question are referred 


60 characters alpha 


12. 


NAME3 


Name of person/office to 
whom questions are referred 


Unlimited alpha 


13. 


TOPIC 


Nature/topic of questions 


Unlimited alpha 


14. 


TRENDS 


Trends in nature of questions 


Unlimited alph/num 



Appendix E: Software Digest 

From Philip Verghis' Help Desk FAW: The definitive globala resource for helpdesks, custom 
relationship management and technical support, Section III Version 5.10, November 1999© 
Verghis 1994-1999 

Table 1 - Commercial Products 



Vendor 


Product 


Platfor 

m 


2020 Solutions 


CalTraker 


PCs~ 


Advantage kbs 


IQ Support Pro 


Various 


Alexander Frances 


MasterSoft OSM 


Various 


Systems Consultants System-4 


Allen Systems Group ASG-Impact 


PCs 


Applied Innovation 


HelpDesk 


Various 


Management 


Expert 


Applix 


Applix 

Enterprise 


Various 


Apsylog 


Apsylog help 
Desk 


Various 



Contact information 

http://wvtfw.2020solutions.org/ 

http://vtfvtfw.akbs.com 

http://vtfVtfw.afsc.co.za/ 

(link not working - July 1, 1999) 
http://vtfVtfw.allensvsqroup.com 

http://vtfvtfw.aim-helpdesk.com/ 

http://vtfvtfw.applix.com 

http://vtfVtfw.apsvlog.fr 

(link not working - July 1, 1999) 
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Ascend Consultancy 


CATS 


PCs 


Astea International 


Heat 


PCs 


Atlanta On-Line 


Help Desk 


IBM 


Systems 


Application 

Suite 


Mainfram 

es 


Automation Centre 


Support Tracker 


Lotus 

Notes 


Baron Software “ 


Manage-it! Help 


PCs 


Services 


Desk 


Bendata Management 
Systems 


HEAT 


PCs 


Blue Ocean Software 


Track-it! 


PCs 


Bridgehead Software 


Bridgehead 
Service Desk 


Various 


Bullseye Systems 


HelpDesk 1 


PCs 


BusinessLine Corp. 


BusinessLine 


Various 


Caliburn Technology 


Caliburn Call 
Center 


Lotus 

Notes 


Clarify Inc. 


ClearSupport 


Various 


Coastal 

Technologies 


HelpIDesk 


PCs 


Commonsense 


Dispatch 


Lotus 

Notes 


Computer 

Associates 


CA-Netman 


Various 


ComputerWorks 


InterTrac Help 
Desk 


Lotus 

Notes 


ConSol* Software 


CallManager 


Java 

based 


Coral Sea Software 


Sysman 


PCs 


Core Technology 


Persist 


PCs 


CustomerSoft 


Li@son 


Various 


CyberSource 


FAULT 


PCs 


Datawatch Corp. 


Q-Support 


PCs 


Decisif Software 


Call Tracking 


PCs 


Solutions 


Software (LSA) 


DKSystems Inc. 


DKHelp Desk 


PCs 


DP Solutions 


In-Vision 


Various 


DVSD International 


Napoleon 


Various 


Envisage Systems 


Envisage 
Support Centre 


PCs 


Epicor Software 


Clientele 


PCs 



(England) 

0784 431 756 
http://www.astea.com 

http://www.atlonline.com/aos/index.htm 



http://www.acentre.com 



http://www.bssinc.com/ 



http://www.bendata.com 

http://www.blueocean.com 

http://www.bridqeheadsoftware.com/ 

http://www.bullsevesystems.com/ 

http://www.bl-corp.com 

http://wvtfw.caliburntech.com 

http://wvtfw.clarify.com 

http://vwtfw.coastaltech.com 

http://vwtfw.qroupapps.com/ 

http://vwtfw.cai.com 

http://vwtfw.computerworks.com 



http://vwtfw.consol.de/Produkte/ 

http://vwtfw.coralsea.com.au 

http://vwtfw.ctc-core.com/ 

products/persist.html 

http://vwtfw.customersoft.com/ 

http://vwtfw.cvber.com.au/cvber/product 

htm 

http://vtfvtfw.datawatch.com 

http://vtfvtfw.decisif.com 

(link not working - July 1, 1999) 

http://vtfvtfw.dksystems.com 

http://vtfVtfw.dpsol.com 

http://vtfvtfw.dvsd . net/P rod ucts/Napoleon 

leon.htm 

http.7/vtfVtfw.envisaqe.co.nz 

http://vtfVtfw.epicor.com 
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Firstwave 

Technologies 


TakeControl 


PCs 


FG&A 


Probe 

Service 


PCs 


Foresight Software 


Management 

System 


Various 

Lotus 

Notes 


GroupApps 


Dispatch 


GroupSoft Systems 


GroupSoft Help 
Desk 


Lotus 

Notes 


GSx Groupware 
Solutions 


GSX Help Desk 


Lotus 

Notes 




GWI 


Lotus 

Notes 


GWI Software 


Collaborative 
Front Office 


Help Desk 


HelpSTAR for 


PCs 


Technology 


Windows 


Hewlett Packard 


HP OpenView - 
ITSM Operation 


Various 


Infinite Access 


CITRIS 


Various 


Infra Corporation 
(Formerly Help Desk 
Systems) 


Infra-Help 


Various 


Integral Solutions 
Corporation 


Support Wizard 


Various 


Ixchange, Inc. 


Customer 

IXCHANGE 


PCs 


Kemma Software 


Bridge 


Various 


KnowledgeSoft 


KnowledgeDesk 


Lotus 

Notes 


Metrix 


OpenUPTIME 


Various 


MGV America 


HelpDesk 


PCs 


Molloy Group 


Top of Mind 


PCs 


Monarch Bay 
Software 


HelpTrac 


PCs 


Motive 

Communications 


Motive Duet 


Various 


Multima Corporation 


NetKeeper 

HelpDesk 

ServiceDesk 


PCs 


Network Associates 


(bought out 

Magic 

Solutions)) 


Various 


Nocom AB 


S.O.S. Help 
Desk 


Various 


Octane Software 


Octane99 


Various 



http://www.firstwave.net 

http://www.fqa-software.com 

http://www.foresight-esp.com/ 

http://www.groupapps.com 

http://www.gsft.com 

http://www.gsx.net 

http://www.gwisoft.com/ 

http://www.helpstar.com 

www.openview.hp.com/itsm 

http://www.bullsevesystems.com/ 

http://www.infra.com.au 

http://www.supportwizard.eom/fspage.h 

http://www.ixchange.co.za/ 

http://www.kemma.com 

http://www.knowledgesoft.com 

http://www.metrix-inc.com/ 

http://www.mgv.com 

http://www.mollov.com 

http://www.helptrac.com 

http://www.motive.com 

http://www.netkeeper.com 

http://www.nai.com/products/helpdesk/ 

esk.asp 

http://www.nocom.se 

http://www.octanesoftware.com 



Opis Corp. 


Support Express PCs 


PHD (Professional 
Help Desk) 


PHD 


Various 


Peregrine 


Service Center 


Various 


Platinum Technology 
(Bought out by 
Computer 
Associates) 


AutoAnswer 

(formerly 

Apriori) 


Various 


PosAm Bratislava 


Help Desk 


Lotus 

Notes 


PRD Software Pty 
Ltd 


HelpMasterPro 
Solution Builder 


PCs 


Primus 


Solution 

Publisher 


Various 


ProAmerica Systems 


Service Call 
Management 


Various 


Quintus Corp. 


HelpQ, 

CustomerQ 


UNIX 


Remedy Corp. 


Action Request 
System 


Various 


Repository 


Customer First 


Various 


Technologies 


Software 


royalblue 


FrontOffice and 


Various 


technologies 


HelpDesk 


SIAL Software 


Fireman 


PCs 


Scopus Technology 


Scopus 


UNIX 


ServiceSoft 


WebAdvisor 


Various 


Service Data 


ServicEdge and 


Various 


Management 


TAC 


Silknet Software 


Silknet 


Various 


SIO Technologies 
Softopia 


MultiHelp 
iTrack and 
iT rack 


OpenVMS 

PCs 


Development 


Enterprise 


Software Artistry 
(now IBM/Tivoli) 


Expert Advisor 


PCs 


Sunrise Software 


Sunrise 


PCs 


Synchrony 

Communications 


Synchrony eRM 


Various 


Syntactica 


Log-Trak 


PCs 


Techflow 


Enhanced Help 
Desk 


Lotus 

Notes 



http://www.opis.com 

(link not working - July 1, 1999) 

http://www.prohelpdesk.com 

http.V/www.pereg rine.com 

http://www.platinum.com/products/sys 

.htm#h 



http://www.posam.sk/ 

http://www.prd-software.com.au 

http://www.primus.com 

http://www.proam.com/ 

http://www.quintus.com 

http://www.remedv.com 

http://www.custfirst.com 

http://www.rovalblue.com 

http://www.sial.com.br 

http://www.siebel.com 

http://www.servicesoft.com 

http://www.sdm1 .com/ 

http://www.silknet.com 

http://www.sio.com/ 

http://www.softopia.com/ 

http://www.softart.com 

http://www.sunrisesw.com 

http://www.svnchronvinc.com 

http://www.svntactica.co.za 
(link not working - July 1, 1999) 

http://www.techflow.com 
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TEK-TOOLS 


jTicket (for 
Remedy) 


Various 


http://www.tek-tools.com/iticket 


TJ Tieto Group 


TJ Help Desk 


Lotus 

Notes 


http://www.tigrouD.com 


United System 
Solutions 


SystemCare 


Lotus 

Notes 


http://www.ussinc.com 


Tele-Support 

Software 


Tele-Support 

HelpDesk 


PCs 


http://www.resource-dvnamics.com 


Tower Concepts 


Razor 


UNIX 


http://www.tower.com 


Tree Tools 


HelpDesk-3 


Various 


http://www.treetools.com.br 


Troika Software 


RESOLVE IT! 


Various 


http://www.troikasoft.com/ 


UniPress Software 


Footprints 


Various 


http://www.unipress.com/footprints/ 


Utopia Technology 
Partners (bought out 
by RoyalBlue) 


Utopia/HelpDes 

k 


PCs 


http://www.utosoft.com 
(link not working - July 1, 1999) 


Vantive Corp. 


Vantive 


UNIX 


http://www.vantive.com 


Vision4 


Gemini 


PCs 


www.vision4.co.uk 


WebWonderland 


WonderDesk 


Web 


http://www.wonderdesk.com 



Aberdeen Group 
Forrester Group 



Gartner Group 



Giaa Group 



IDC 



publishes a detailed review of customer service software 
each year. 

analyzes and predicts the impact of technology change 
on large companies, consumers, and society, 
has some reports on prominent vendors and compares 
them. Their reports gives you an idea about the market 
and puts the larger vendors in an 'Ability to Execute vs. 
Completeness of Vision' grid. 

their research focus is on helping companies integrate 
their businesses with the Internet, 
is a comprehensive resource on worldwide IT markets, 
trends, products, vendors, and geographies, 
focuses exclusively on how the Internet and other 
Jupiter Communications technologies are 

changing traditional consumer industries, 
has reports titled: 

• "Automating Your Support Center - A Practical 
Guide to Assessing Service 

• Automation Tools" 4th edition. 

"A Field-Tested RFP Template - For Selecting a 
Customer Response Center Automation System" 
(includes an Excel spreadsheet form) 

• "Implementation Lessons from Leading-Edge 
Users" 

• "Creating A Support Web Site - The Process, 
Potential and Pitfalls." 

Meta Group has done research on Help Desks and Call Centers. 



Knowledge Networks 
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The Muns Group - 

contact Lon (prior to forming Renaissance Partners) published a 

Hendrickson at booklet titled: "Selecting Customer Support Technology, 

webmaster@bendata.co 23 Important Considerations" . 
m. 



National Software 
Testing Labs 



reviewed some help desk products. 



Ovum 



SPEX (Software 
Product Expertise) 



Unified Business 
Solutions 



is an independent research and consulting company 
offering advice on IT and telecommunications markets. 
They have a regularly updated report titled 
"Ovum Evaluates: Help Desk and Customer Support 
Systems." 

is an IT research firm that provides comprehensive 
software package evaluations to assist global companies 
in the selection of critical applications. The reports can 
be downloaded from their site. Site visitors can obtain 
more than 20 free sample reports from the kits. 

has a web-based software selection tool. 



Yankee Group 



focuses on strategic planning assistance, technology 
forecasting and IT industry analysis. 



Review by a reader of the help desk list (suitable for smaller organizatio 
Related Tools 

Email handling tools 



@Once 

http://www.onceinc. 

com/ 

Acuity 

http://www.acuitv.co 

m 

Adante 

http://www.adante.c 

om 

Aditi Corporation 

http://www.talisma.c 

om 

Brightware 

http://www.briqhtwa 



is an email management services company that helps 
Fortune 1000 companies manage inbound customer email 
and outbound email marketing. 

offers the WebCenter product line, which offers web-based 
customer interaction solutions. 

offers an enterprise-class customer e-mail management 
solution that allows companies to efficiently manage and 
respond to hundreds or thousands of customer e-mails daily, 
offers Talisma - an easy-to-use, out-of-the-box customer e- 
mail and Web-form manager that helps sales and service 
teams respond quickly and effectively to electronic 
communications flooding busy aliases, such as 
info@companv.com 

provides customer assistance software on the Internet. 



re.com 



eGain 

http://www.egain.co 

ni 

Island Data 
http://www.islanddat 
a. com 
Kana 

Communications 

http://www.kana.co 

m 

mailQueue . 

http://www.mailque 

ue.com 

Calypso Message 
Center 

http://www.mcsdalla 

s.com 

Mustang Software, 
Inc. 

http://www. mustang 
.com 



offers the eGain Email Management System 



automates the process of reading and responding to 
electronic support requests. 

offers the Customer Messaging System that provides robust 
enterprise-class tools to manage customer e-mail 
communication. 

offers a web-based e-mail management solution allowing 
companies and their customers access to high performance 
tools when managing their corporate e-mail. 

is designed to efficiently process, route, track and manage 
inbound departmentalized e-mail. 



offers the Internet Message Center designed to handle 
inbound e-mail, from web sites where the e-mail is being 
directed to a department, rather than an individual. 



Integrated suites (sales, marketing etc. — often called "Customer Relationship" tools) 

• Aurum Software, at http://www.aurum.com offers a complete set of 
integrated sales, configuration, and call center components. 

• Baystone Software, at http://www.bavstone.com offers software that 
enhances communication between the sales, marketing, support, and 
quality functions in your organization. (Note: Baystone has acquired by 
Remedy Corporation.) 

• CorepoInt, an IBM company, at http://www.corepoint.com offers customer 
relationship management software. 

• Chordlant Software, at http://www.chordiant.com/ delivers a complete 
application environment for consumer transactions through call centers, the 
internet, point-of-sale locations. 

• Slebel, at http://www.siebel.com/ provides a comprehensive and scalable 
customer information system 

Knowledge bases and self-help tools: 



Acuity 

http://www.acuity.com/ 
Acme Software 
http://www.acmesoft.com 

Advantage kbs, Inc. 
http://www.akbs.com 



offers online self-help capabilities and real-time 
multimedia communication functionality, 
offers FAQtory, a dynamic, interactive Q&A hosted 
internet application 

offers a suite of products to automate problem 
resolution at the support center or at the end- 
user's desk. 



Courion Corporation 
http://www.courion.com 

The Haley Group 

http://wwV'/.halev.com/help 

lcpr.html 

Inference 

http://www.inference.com 
Knowledge Broker, Inc 
http://www.kbi. com/ 

Knowlix 

http://www.knowlix.com 

MetaQuest Software 
htip://www. metaguest. co 
m/ 

The Molloy Group 

http://www.knowledgebrid 

ge.com 

NetHelp International 
http://www.nethelpnow.co 

m 

Platinum Technology 
(now part of Computer 
Associates) 

http://www.platinum.eom/p 

roducts/dataw/appisres.ht 

m 

Primus 

http://www.primus.com 
Right Now 

http://www.rightnowtech.c 

pm 

ServiceSoft 

http://www.servicesoft.co 

in 

ServiceWare 

http://www.serviceware.co 

m 

Also see their knowledge- 
based support site at: 
http://www.rightanswers.c 
om 



offers Internet-based self-service applets which 
automate common support tasks - integrated with 
help desk systems to track support activity. 

allows you import or create custom case-bases using 
the CPR C++ class library. 

offerst the k-commerce suite of products 

offers "Experience Bases" you can purchase to pre 
populate your knowledge base, 
has a series of knowledge management products that 
integrates seamlessly with various support 
management tools. 

offers Triage - a diagnostic software environment and 
Census - a defect tracking system. 

offers Knowledgebridge - a tool that ties into existing 
call tracking systems. 

offers a heuristic ("self-learning") knowledge base, call 

logging and reporting 

software. 



has Apriori Plus that can tie into various call tracking 
tools. 



offers a series of knowledge management tools 



Right Now Web 



offers a complete Internet Customer Support solution 



offers multimedia "Knowledge-Paks" you can purchase 
to pre populate your knowledge base. 




has a web based data collection system that improves 
South Wind Design the productivity of your technical support staff, help 

http://www.swdi.com desk or call tracking system by using a browser and 

Active X. 



Verity 

http://www.verity.com 
Steriing 
Software 
http://www.sol 
ve. sterling. CO 
m/dip/ 

Teiamon 
http://www.tel 
amon.com 



offers the Knowledge Retrieval Product Suite 



has SOLVE: Diplomat which takes disparate software systems and 
makes the data logical 

and easy to use across the entire network enterprise. 

has a paging solution called "TelAlert" that runs on UNIX and 
Windows NT and integrates with popular help desk products. 



ACDs and VRUs 



• ACDs - can route incoming calls to the next available Help Desk staff 
member, route incoming calls, and can collect data on the calling patterns. 
These typically reside in the phone system switch. Highly recommended if 
you have a medium to large staff and call volume. This will allow you to 
measure how many calls are 'abandoned' (i.e. customer hangs up before 
speaking to an agent), 'average time to answer' (i.e. how long it take you to 
pick up the phone) etc. 

• VRUs -- Can filter and route calls based on information about the caller or 
problem. With CTI (Computer Telephony Integration), you can pre populate 
your 'trouble ticket' with your client's information. VRUs can be used for 
password resets etc. 

• If you are interested in learning more about this and related areas, check 
out 

Call Center magazine . 

Sample companies mentioned in the above two categories... 

ACDs 



• Aspect CallCenter System at www.aspect.com has staffing software 
(TeleCenter System) available from the TCS Management Group which 
automates many of the tasks associated with managing a call center 
workforce. 

• AT&T at http://www.att.com/ has Definity Communications Systems has the 
Definity and Conversant product lines. 

• Northern Telecom at http.7/www.nortenetwori<sl.com has the ACD Max line. 

• Teledata Solutions at http://www.tdata.com/ has Call Link. 

• Teloquent Communications at http://www.teloquent.com has the Distributed 
Call Center. 



VRUs 

• Many of the same vendors above have VRUs. 

• Talx Corporation at http://www.talx.com 
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